DEPARTMENT OF CULTURE AND THE ARTS FUNDING … · o#ers a complete map of the process of public...

DEPARTMENT OF CULTURE AND THE ARTS

FUNDING PROCESS REVIEW - STAGE 2

PUBLIC VALUE MEASUREMENT FRAMEWORK

MAY 2012

DISCLAIMER

This report has been prepared for the Department of Culture and the Arts. The information contained in this report has been prepared with care by the authors and includes information from apparently reliable secondary data sources which the authors have relied on for completeness and accuracy. However, the authors do not guarantee the information, nor is it intended to form part of any contract. Accordingly all interested parties should make their own inquiries to verify the information and it is the responsibility of interested parties to satisfy themselves in all respects.

This report is only for the use of the party to whom it is addressed and the authors disclaim any responsibility to any third party acting upon or using the whole or part of its contents.

Document Control

Document Version Description Prepared By Approved By Date Approved

v 1.0 Funding Process Review - Stage 2: PVMF Michael Chappell 1 May 2012

PERTH: 23 Lyall Street South Perth Western Australia 6151 • t (08) 9367 1511 • f (08) 9367 4066MELBOURNE: Cannons House Level 7, 12-20 Flinders Lane Melbourne Victoria 3000 • t (03) 9654 5775

e [email protected] • www.pracsys.com.au

i

CONTENTS

1 EXECUTIVE SUMMARY 1

1.1 Theoretical outline of PVMF model 1

1.2 Key Findings 2

1.3 Summary of recommendations 4

2 INTRODUCTION 5

2.1 Project Background 5

2.2 Report Structure 5

3 PUBLIC VALUE MEASUREMENT – TOWARDS A BETTER UNDERSTANDING 7

3.1 Project Purpose 7

3.2 DCA’s Focus on Public Value 7

3.3 A Universal Challenge 8

3.4 Whole of Government 10

3.5 Understanding Cultural Value 10

3.6 Instrumental Value 11

3.7 Intrinsic Value 11

3.8 Institutional Value 12

3.9 Total Public Value 12

3.10 Artistic Quality Debate 12

3.11 The Arts and Creative Industries 13

4 CURRENT DCA MEASUREMENTS 15

4.1 DCA Funding Programs 15

4.2 Application, Assessment and Acquittal Process 15

4.3 Metric Inventory 16

4.4 Financial and Quantitative Harmonised Reporting 17

4.5 DCA Key Performance Indicators 17

4.6 KPI Review 18

ii

Department of Culture and the Arts: PVMF

5 A NEW APPROACH – THE PUBLIC VALUE MEASUREMENT FRAMEWORK 19

5.1 Towards a sharper system 19

5.2 Public Stakeholder Sessions with funded arts and cultural organisations 20

5.3 Peer Review 21

5.4 Opportunities for co-investment 24

5.5 Practical Testing of the Value Framework 25

5.6 A Public Value Measurement Framework Model 25

5.7 How do the elements of Public Value fit together? 28

5.8 Defining Process Reform 28

6 STRATEGIC ALIGNMENT OF METRICS TO POLICY, GOALS AND DECISION RULES 31

6.1 Policy and Measurement Principles 31

6.2 Distinguishing and aligning policy choices and measurement decisions 32

6.3 The ongoing development of DCA policy and the PVMF 34

7 TESTING THE VALUE FRAMEWORK – PEER REVIEWS AND FEEDBACK 37

8 RECOMMENDATIONS AND IMPLEMENTATION ISSUES 41

8.1 Recommendations 41

8.2 Implementation Challenges 42

APPENDIX 1: CONTINGENT VALUATION METHOD 47

APPENDIX 2: METRICS BY DCA FUNDING PROGRAM 49

APPENDIX 3: DCA STRUCTURE AND DATA 51

APPENDIX 4: MEETINGS AND CONSULTATION 53

APPENDIX 5: MINUTES 55

APPENDIX 6: FORM A B 57

APPENDIX 7: ISSUES PAPER 1 AND 2 59

1

1.0

1 EXECUTIVE SUMMARY

The Department of Culture and the Arts (DCA) is seeking to better understand and measure the public value created by its investments in arts and cultural activity in WA and their role as a development agency for the sector, through development of a new public value measurement framework (PVMF).

1.1 THEORETICAL OUTLINE OF PVMF MODEL

Australia, European and American arts policy makers have long discussed issues around excellence, access, reach and the wider impact of the arts, and have struggled to make decisive progress or innovations in defining and measuring key outcomes in a way that actively shapes funding practice and activity.

The temptation with any measurement system is to focus on what is easier to measure (outputs) rather than what it is important to measure (outcomes). To develop the measurement framework, a shared understanding of what is meant by the terms quality, reach, impact and value was generated through research and consultation, in order to develop agreed definitions for both the outcome areas and policy drivers/goals.

Value has a layered and complex meaning. It has a strict narrow economic meaning, a more fluid intrinsic value and a longer-term institutional value in addition to the ‘externality’ value that creates skills and experiences vital to the creative industries. Research found that value has a broad community meaning that can be measured using contingent valuation. The theory behind the PVMF model is that all of these elements should be considered together to gain a full picture of the true meaning of value. The challenge in developing the metric

framework is to develop definitions of quality and reach that will allow an aggregate intrinsic utility calculation.

The PVMF calibrates with contemporary thinking on public value measurement. Broader cultural value insights from the literature have been interpreted to develop a practical and useful system of public value measurement for everyday use by DCA in policy development, funding application assessments and funding acquittals. The clear intention is to produce a framework that generates an evidence base to assist funding decision-making processes and create a foundation of evidence to argue for more (and more diverse) funding for the sector.

The model involves improvements to both the definitions of terms and methods of data collection to enable measurement of differences or similarities between what the funding applicant and peers thinks of their work, comparisons between the activities of the same artist/art form over time and comparisons between funding programs and activities from different art forms. It could also be used more widely as a benchmarking tool with funding programs in other jurisdictions across Australia.

In contrast with the existing application-assessment system that gathers data about the funded activity before it happens, the PVMF tool is intended as a learning system that will provide the opportunity for both the funding applicant and the peer assessor to review their assessment after the event. The analysis of variance on their respective assessments will give each the opportunity to learn how the output was different from their expectation, enabling better judgement over time.

2


A vital element in the post-output assessment is the view of the public. The PVMF tool must include an integrated online system allowing the public to register their feedback about the quality and impact of the funded output when they ‘consume’ it, to ensure that data collection is seamless, uniform, inexpensive and continuous.

Any effective measurement framework needs to integrate policy, the measurement definitions, and the valuation and funding process. The measurement tool must have distinct links back to policy and funding decisions so that the framework is a dynamic informer of an evolving public value contribution.

1.2 KEY FINDINGS

As arts funders globally have struggled with measuring key value outcomes, development of the PVMF measurement model is mindful of the need to produce something that can be used successfully by other arts funders.

Peer review of the PVMF model has been universally positive, finding that the framework offers a complete map of the process of public funding of culture from decision-making through to the assessment and articulation of value. It is felt that DCA has made progress in developing a strategy for maximising the public value of culture that has not occurred within any other funding body internationally. The proposed framework is considered simple without being reductionist and is expected to lead to greater transparency, understanding and trust between DCA, clients and other stakeholders.

Within the current funding climate, it is clearly recognised that there is a need to articulate the value of culture using methods that fit

with central government’s decision-making, rather than simply demonstrate the value through the lens of economic or social ‘impact’. Value measurements highlight the scale of the impact of funded activities relative to the funding required to achieve them.

While the usefulness of capturing value-creating impacts has been widely supported, the project has identified the difficulties involved in doing it well and the costs involved. Arts and cultural departments are frequently criticised for not pursuing a rigorous research agenda to capture the impact of the arts, yet are also criticised if spending on administration and overheads diminishes the proportion of public funds flowing directly to arts organisations.

The alignment of the policy and measurement definitions must produce an evaluation, funding and acquittal process that produces robust, credible and transparent outcomes. Numerous testing sessions with artists and cultural leaders have generated metrics to effectively capture the practice and intentions and satisfy the desire of the funder to better account for the outcomes and value produced. It is fundamental that outcomes from the funding and evaluation process feed back into the policy decisions and future resource allocation rounds.

Essential characteristics of a robust measurement system have been identified and developed throughout the process, and include:

• Simplicity: Due to a complex array of variations across art forms and outcomes, the measurement system needs to be simple enough for the entire sector to use it, displaying uniformity and linking back to the policy

3


• Standardisation: The same set of parameters must be used to ensure a common understanding of what to assess and enable comparison across various programs or art forms

• Documented Decision Rules: Decision rules provide a framework and accountability for decision making, ensuring the integrity of the measurement framework and providing a feedback mechanism to DCA policy

• Evidence base: An effective measurement framework must be evidence based so that the integrity of decisions made can be ensured. Over time, an effective measurement framework generates a record of the evidence base that reinforces the capacity of the system to ‘learn’

• Feedback loops: Incorporating feedback loops enables the performance of organisations and projects to be measured, with results informing ongoing decisions about further funding and policy. The closed loop provides the evidence upon which future decisions can be made and provides the record of learning events to reinforce the integrity of the system.

Despite the challenges associated with developing a widely approved, robust, reliable and comprehensive tool, the funding process review has identified numerous benefits for DCA associated with adopting a public value measurement framework. These include:

• Improvement in the quality of DCA’s decision making and return on investments

• Improvements to the accountability and transparency of DCA’s activities as a development agency and public funder, increasing the public value of all operations

• The ability for DCA to shape an overall portfolio of investments that will better deliver on their strategic aims and ambitions

• The ability for DCA to become a more effective development agency and funder

• The creation of a stronger shared vision of success between DCA and the WA arts sector

• Strengthening of the credibility and impact of the arts evidence base and investment case for the arts

• The opportunity for DCA to engage with the arts sector to produce a measurement framework that sensitively reflects their understanding of how best to foster and measure artistic quality, engagement and innovation

• The ability for DCA to develop an objective measurement system with multi-point subjectivity that will generate a higher level of trust and support from the arts sector and other partner agencies

• Stimulation of debate with other partner investors and key stakeholders across Australia about how the policy and practice of arts funding can be improved

• The opportunity for other funders to more closely align their investment and measurement approaches with those of DCA

4


1.3 SUMMARY OF RECOMMENDATIONS

The PVMF poses a range of operational challenges for the DCA, which are expected to be identified, discussed and resolved during the next stages of the funding process review.

It is recommended that DCA generate a project plan and consider resourcing to implement recommendations arising from this stage of the process. Action items include:

1. Classification of core component definitions of the adopted public value model, including outcomes, quality, reach and impact

2. Adoption of a one-dimensional scale for measuring each parameter

3. Continued review of DCA’s policy frameworks to identify gaps and ensure that priority output areas are producing the full range of intended value

4. Review of Key Performance Indicators, ensuring they include core public value metrics

5. Creation of a set of weightings reflecting DCA’s policy intent for each funding program

6. Aligning funding application and acquittal forms to the new model

7. Consideration of options for a public value training program within DCA to ensure definitions are applied uniformly

8. Provision of an effective communication strategy for public value awareness amongst key stakeholders and relevant interest groups

9. Exploration of how funding, acquittal and assessment processes can be run effectively and efficiently online and through related media formats within two years

10. Creation of a database of pre-output (funding assessment) and post-output (funding acquittal) data that can be used for annual, aggregate public value assessments

11. Creation of decision rules for the funding allocation process to be specified in the policy framework

12. Development of internal and external opportunities for longitudinal research and evaluation based on core data within the framework

13. Harnessing of the PVMF to a clear Gross Value Added (GVA) model

Finally, it is recommended that DCA develop a clear strategy for investing in the wider public value infrastructure that will allow it to capture broader public value, including educational attainment, social and community benefit, place making and wider economic benefits.

5

2.0

2.1 PROJECT BACKGROUND

The Department of Culture and the Arts (DCA) operates under the provisions of the Government of Western Australia’s Public Sector Management Act 1994. The DCA leads in the alignment of the Culture and Arts Portfolio to the State Government’s policy and management framework through the minister for Culture and the Arts. The DCA also works with a range of stakeholders to coordinate arts and cultural policy development, distribution of funding, implementation of national and strategic projects and specialised research.

In 2010, DCA released two new guiding documents, the 2010-2014 Strategic Plan and the 2010-2014 Creating Value: an Arts and Cultural Sector Policy Framework. Each document emphasises the delivery of public value as the driving principle of the Department’s funding program.

Through the Development and Strategy Directorate (DSD), the Department provides funding support to the arts and cultural sector to 49 key organisations and offers grants programs across a range of disciplines.

Following the release of the Strategic Plan and the Creating Value framework, DSD has initiated a range of strategic projects to give effect to its public value priorities. The goal of the projects is to provide the Department with a stronger evidence base for the funding decisions it makes based on the principle of best public value. The evidence base is intended to include a rationalisation of existing measures and new measures to assess the public value delivered through the funded arts and cultural program.

The stated objective of the project is to identify a methodological framework to provide workable data in terms of measures and indicators, including methods of data collection that can be used to evaluate the range of public value outcomes delivered by the Department and its clients. These public value outcomes are to include the four public value dimensions of quality, reach, impact and value.

The purpose of the Funding Process Review is to assess how DSD can optimise the public value delivered by its funding activities and programs. The Funding Process Review involves three stages:

Stage One: Efficiency of funding program processes and administration (completed 2010)

Stage Two: Public Value Measurement Framework (this project)

Stage Three: Alignment of Funding Processes, Programs and the Public Value Measurement Framework to the Creating Value Framework.

2.2 REPORT STRUCTURE

The report is structured to give the reader an overall understanding of the state of contemporary thinking about public value measurement in the arts and cultural sector – particularly as it relates to the current state of performance measurement at the Department of Culture and the Arts. This is a backdrop to the new system proposed for DCA, which derives from the Creating Value framework – enhanced by best practice thinking on cultural value and measurement techniques from around the world. Finally, the report examines some of the practical implementation issues that DCA will face when it decides to apply the PVMF to

2 INTRODUCTION

6


its current processes in policy development, funding applications and funding acquittals. The Chapter headings are as follows:

Chapter 3: Public Value Measurement – towards a better understanding

Chapter 4: Current DCA Measurements

Chapter 5: A New Approach – the PVMF

Chapter 6: Strategic Alignment of Measures to Policy and Decision Rules

Chapter 7: Operationalising the Framework within a State Arts Funding Agency

Chapter 8: Testing the Value Framework – Peer Review and Feedback

Chapter 9: Conclusions and Recommendations

To reflect the sheer volume of consultation and analysis undertaken for the project, the report concludes with a series of technical appendices as follows:

• Contingent Valuation Method

• Metrics by Program Name

• DCA Structure and Data

• Meetings and Consultation Minutes

• The proposed data collection structure for measuring intrinsic quality and reach dimensions (Form A and B)

• Issues Papers 1 and 2

7

3.0

3.1 PROJECT PURPOSE

The purpose of this project is to develop a new measurement framework to capture the value created by arts and cultural activity in WA. DCA is seeking to better understand and measure the public value created through its investments in arts and culture, and their role as a development agency for the sector. The consultant team worked with DCA to identify the key outputs, outcomes and value created through public investment in arts and culture. This has led to modelling the links between the policy framework and the measurement framework proposed.

The resulting measurement framework has been benchmarked against leading edge thinking on the ways in which a public funder might measure the value of its cultural investments. This ensures that the measurement framework DCA will be using reflects best practice understandings around definitions and measurement techniques.

As a consequence of this approach, the proposed measurement framework has two important users. First, DCA, as it attempts to harness the measurement framework in investment activities and second, the broader arts and cultural sector in their efforts to measure the value contribution of the sector to the public at large. In producing the framework, DCA have been challenged to bring clarity to their intentions and ambitions and it is expected that the measurement framework will provoke similar challenges to the broader arts and cultural sector in Australia and elsewhere.

3.2 DCA’S FOCUS ON PUBLIC VALUE

DCA’s explicit focus on public value was established in DCA’s Creative Value policy framework, which presents DCA’s starting point in terms of public value measures and outcome areas. As Figure 1 summarises, DCA has developed four key public value measures – quality, reach, impact, and value. In turn, these four key measures are the foundation of the measurement model we have produced.

As the project has progressed, the pay-offs for DCA of adopting a public value measurement framework have become clear, namely to:

• Improve the accountability and transparency of DCA’s activities as a development agency and public funder, increasing the public value of all it does

• Improve the quality of DCA’s decision making, and return on its investments

3 PUBLIC VALUE MEASUREMENT – TOWARDS A BETTER UNDERSTANDING

8


Figure 1: DCA Public Value Measures

Quality:Quality is seen as a measure of creative process and product and will include the distinctive, innovative and significant elements of the creative experience. Quality will be measured through a combination of self assessment, peer assessment, audience and public response.

Reach:Reach is seen as a measure of the access to and participation in arts and cultural activities. It will measure the breadth and depth of engagement, through attendance and participation data alongside audience and public satisfaction with their level of engagement.

Impact:Impact is seen as a measure of the social, cultural and economic impact of arts and cultural activities, and will include the transforming impact through engagement. Impact will be measured through the quantitative and qualitative review of outcomes from engagement.

Value:Value is seen as a measure of both the economic value of arts and culture and the appreciation of arts and culture in the Western Australian community. Value will be measured through both the return on investment and the value of culture and arts through community surveys.

Outcome Areas:

Results across the four Strategies will be aggregated to demonstrate the delivery of public value in the four Outcome Areas: Creative People; Creative Communities; Creative Economies; Creative Environments.

Outcomes in each of these areas will contribute to the DCA Public Value Principles: Creativity and Engagement.

Measurements will be drawn from a number of existing and newly designed surveys and data sources. These will assist in determining the public value of arts and culture to Western Australia

• Allow DCA to shape an overall portfolio of investments that will better deliver on their strategic aims and ambitions

• Allow DCA to become a more effective development agency and funder

• Help create a stronger shared vision of success between DCA and the arts sector in WA

• Strengthen the credibility and impact of the arts evidence base and investment case for the arts

• Provide DCA with the opportunity to engage with the arts sector to produce a measurement framework that sensitively reflects their understanding of how best to foster and measure artistic quality, engagement and innovation

• Enable DCA to create an objective measurement system, with multi-point subjectivity, that will generate a higher

level of trust and support from the arts sector and other partner agencies

• Stimulate debate with other partner investors and key stakeholders across Australia about how the policy and practice of arts funding can be improved

• Provide the opportunity for other funders to more closely align their investment and measurement approaches to those of DCA

There are significant prizes for both funder and funded across the sector in Western Australia.

3.3 A UNIVERSAL CHALLENGE

In securing these benefits from a new model, DCA required issues to be addressed that represent universal challenges for all public funders and investors of arts and culture. Australia, European and American arts policy makers have long been discussing issues

9


around excellence, access, reach and the wider impact of the arts. Therefore, the measurement model was developed with the need to produce something that could be used successfully by other arts funders.

This is important given that funders have struggled to make decisive progress or innovations in defining and measuring the key outcomes in a way that actively shapes funding practice and activity.

Since the 1980s the value of the cultural sector has been demonstrated through the lens of ‘impact’, whether economic or social. However in recent years there has been recognition, both within central government and in parts of the publically funded cultural sector, of the need to more clearly articulate the value of culture using methods that fit in with central government’s decision-making. This is an important distinction, because it highlights the scale of impact of funded activities relative to the funding required to achieve them. This is particularly important because of the ‘cooler’ funding climate faced by arts and cultural organisations in recent times.

The lack of progress in designing and implementing new approaches cannot be attributed to an absence of informed academic research and practitioner experimentation. There is growing and expansive literature about the ‘art’ of measurement from gross value added (GVA) models, social return on investment models, stated or revealed preference models, intrinsic value measurements including the debate about measuring artistic quality or vibrancy and the artistic experience through

to a wider range of applied practice and measurement in arts and culture interventions.

This project has drawn on a burgeoning expert literature on individual measurement elements of the PVMF1. However, globally, there are no comprehensive models in which expansive definitions of public value outcomes have produced a coherent, leading edge measurement framework that:

• Comprehensively models the value creating impacts of arts and cultural activities and develops a widely supported strategy to measure them

• Is capable of shifting perceptions of the value of arts and cultural investment with public and politician alike

• Provides the necessary clarity of ambition and evidence base that would allow public funders and other investors to make significant changes to the way in which they are shaping their overall pattern of portfolio investments.

In reviewing international practice it is revealed that few, if any, Government arts departments or arts councils have taken upon themselves the responsibility of consistently capturing these broader ‘value’ creating contribution of the arts (embracing not just artistic and cultural value, but economic, social and public value as well).1 Important recent contributions include: O’Brien, D.

(2010) ‘Measuring the value of culture: a report to the Department of Culture Media and Sport,’ DCMS; Bunting, C. (2010) ‘Achieving Great Art for Everyone: A Review of research and literature to inform the Arts Council’s 10-year strategic framework,’ Arts Council England; Bakhshi, H. (2009) ‘Measuring Intrinsic Value – how to stop worrying and love economics’ Mission Models Money; Throsby, D. (2010) ‘The Economics of Cultural Policy’ Cambridge University Press.

10


This is something for DCA and other public agencies in the arts to reflect on very carefully. The question is not just whether it is useful to capture the value-creating impacts, but that public agencies need to recognise the difficulties in doing it well and the department costs involved. Government arts and cultural departments along with arts councils are frequently criticised for not pursuing a rigorous research agenda to capture the impact of the arts and are also criticised more vocally if in making such investments in measurement and evaluation, the amount of money they are spending on ‘administration’ and ‘overheads’ is diminishing the proportion of public funds flowing directly to arts organisations.

The logical conclusion of this is that until funding agencies develop measurement frameworks that actively underpin their investments and which provide a compelling account of both the return on public investment and the wider value being created by arts and cultural activity, that argument won’t be won.

3.4 WHOLE OF GOVERNMENT

DCA contributes to the achievement of the Government’s goal for: ‘Greater focus on achieving results in key service delivery areas for the benefit of all Western Australians’. The Government’s desired outcomes for the culture and arts portfolio are:

• A creative, sustainable and accessible culture and arts sector

• Western Australia’s natural, cultural and documentary collections are preserved, accessible and sustainable

Inherent in these outcomes will be a creative community that participates in and values culture and the arts, contributing to the quality of life and well-being of all people throughout WA. The key performance indicators (KPIs) supporting this are intended to provide the evidence that the resource allocation decisions made by DCA are driving activity towards the outcomes.

These outcome sets are a mixture of cultural, economic and public value. What then are some of the measurement challenges in these areas?

3.5 UNDERSTANDING CULTURAL VALUE

Perhaps unsurprisingly, there is no one accepted definition of cultural value within the expert literature on the subject. Recent reviews of this literature2 have underlined ongoing controversies as to whether cultural and economic value are mutually exclusive categories in a measurement sense, and the extent to which the language of outputs and outcomes can easily be applied to the measurement of cultural value.

We do not intend to rehearse those arguments here and have reviewed them in our previously submitted benchmarking report. Rather, starting with the overall intentions of DCA in terms of their approach to public value, it is clear that John Holden’s well-established formulation – of instrumental value, institutional value, and intrinsic value – is a useful starting point in modelling the public value measurement framework.

2 O’Brien (2010.19) Measuring the value of culture: a report to the Department of Culture Media and Sport, DCMS

11


For Holden3, cultural value can be understood as instrumental value, institutional value and intrinsic value. The three forms of value are interdependent and rely on each other to form an overall picture of cultural value.

Instrumental value is generated by the social and economic policy uses of culture, for example to raise exam results or tackle social exclusion. Institutional value refers to the kind of value discussed by Mark Moore’s4 work on public value, where organisations generate trust or esteem by the way they engage their users. Intrinsic value is that form of value associated with ideas of aesthetic excellence and individual enjoyment.

We think part of the problem in modelling value metrics is that these terms themselves – instrumental, intrinsic, institutional – have become a base for broad debate on the meaning of public value. For the purposes of a robust and transparent funding system, the key is that funder and funded are operating with a shared understanding of what these terms mean, and that the value systems as a whole, and other supporting evaluation activity, are capable of capturing the full value of cultural investments.

In developing the core metrics set, we have therefore developed more tightly focused definitions of these key value categories, to allow for an overall aggregation of the value being created.

3 Holden (2006) Cultural Value and the crisis of legitimacy, London: Demos

4 Moore (1995) Creating Public Value Strategic Management in Government Harvard University Press

3.6 INSTRUMENTAL VALUE

For example, we treat instrumental value as being calculated from the instrumental utility derived from the consumption of the output divided by the investment amount. The best proxy for this is a consumption utility monetary transaction (e.g. ticket purchased, sponsorship provided), which is the product of the tickets sold and the reach. Instrumental value can therefore be expressed as:

!!"# =!!"#

$ !"#$%&'$"&

Other forms of instrumental value, in a Holden type definition, such as wider education or social effects, can be captured in other parts of the metric framework, or wider evaluations.

3.7 INTRINSIC VALUE

We treat intrinsic value generated by the investment as the intrinsic utility from consumption of the output divided by the investment amount. Intrinsic utility is derived from the quality and reach of the output, which is measured using the multi point subjective assessments of quality and reach described in the proposed PVMF assessment system. This is our major departure from Holden. Holden does not think that intrinsic value – including highly subjective quality considerations – lends itself to formal measurement. We set ourselves the challenge in developing the metric framework to develop definitions of quality and reach which would allow an aggregate intrinsic utility calculation, which we discuss in our presentation of the model.

12


Intrinsic value can therefore be expressed as:

!!"# =!!"#

$ !"#$%&'$"&

3.8 INSTITUTIONAL VALUE

Every funded organisation generates institutional value through their activities. We understand institutional value in the same wider public sense that Holden does, where organisations generate trust or esteem by the way they engage their users. Institutional value is often regarded as the cultural or artistic capacity in assets, networks and content handed down from one generation to the next.

By virtue of their size and longevity, some institutions have accrued a store of institutional value that may seem to exceed their instrumental and intrinsic value. Museums, galleries and libraries sometimes rely too heavily on the presumption that they will be funded in future because they have been funded in the past. In other words, that their institutional utility will be sufficient to deliver value for the funding they receive. Like smaller arts organisations, they also need to focus on their instrumental and intrinsic utility by methods of the type specified in this PVMF.

We believe this wider public value is important and needs to be captured through explicit and one-off evaluations of public value. Techniques for measuring institutional value (such as contingent valuations and hedonic pricing models) as outlined in Appendix 1.

Institutional value can therefore be expressed as:

!!"#$ =!!"#$

$ !"#$%&'$"&

3.9 TOTAL PUBLIC VALUE

In aggregate therefore, total public value is the sum of instrumental, intrinsic and institutional value, as expressed in the following term:

!! =!!"# + !!"# + !!"#$$ !"#$%&'$"&

Although the purpose of this project is not to extend the academic debate on public value per se, it is useful that the PVMF calibrates with contemporary thinking on public value measurement. We have interpreted the broader cultural value insights from the literature to suit our purpose – to develop a practical and useful system of public value measurement for everyday use by DCA in policy development, funding application assessments and funding acquittals. In doing so, we acknowledge that the notions of instrumental, intrinsic and institutional value overlap significantly – and that our system design offers a workable compromise. The clear intention is to produce a framework that generates an evidence base that assists funding decision-making processes and assists to create a foundation of evidence to argue for more (and more diverse) funding for the sector.

3.10 ARTISTIC QUALITY DEBATE

There are now an established range of methods to capture the qualitative cultural impacts of public investment, and in turn cultural value. These include self-evaluations, peer and user-review and stakeholder analysis. There is also growing support for ‘artistic self-assessment.’ Bailey and Richardson5 have suggested

5 Bailey, J. & Richardson, L. ‘Meaningful Measurement: a literature review and Australian and British case studies of arts organisations’ conducting artistic self-assessment’ Cultural Trends, No. 76

13


that there is no ‘one size fits all’ approach to assessing what Australasians refer to as ‘artistic vibrancy’ (and what in the UK is referred to as ‘excellence’):

‘The idea that universal templates can be developed across companies and/or artforms is thus belied by empirical evidence, and the literature shows attempts to create such templates often degenerate into box-ticking’.

Methods suggested by their case studies, which include peer and staff opinion, could include assessment panels, guest artist surveys and staff days – reflecting the 360 degree review approach supported by much of the literature on performance management.6

In the UK, Arts Council England is currently rolling out an Artistic Assessment Scheme that is centred on ‘artistic assessments by assessors who have knowledge and experience of the arts’ and focuses on ‘excellence’ rather than ‘impact’.

More generally, the issue of defining artistic ‘excellence’ remains a sticky issue once you step beyond the established approach of judging against appropriate standards as set by professional practice in the relevant artform, with those judgements taken by informed practitioners.7

The Australia Council has already triggered a vibrant debate about how best to approach and define artistic vibrancy8, adopting a wider 6 Selwood, S. et al (2010) ‘John Holden’s Capturing Cultural

Value: How culture became a tool Government Policy’ Cultural Trends

7 McMaster, B. (2008) ‘Supporting excellence in the arts: from measurement to judgement.’ Department of Culture, Media and Sport; and Throsby, D. (2010) ‘The Economics of Cultural Policy’, Cambridge University Press

8 Bailey, J (2009) ‘Meaningful Measurement: A review of the literature about measuring artistic vibrancy,’ Australia Council for the Arts

and richer approach to vibrancy and excellence including within their working definition of artistic excellence; audience engagement and stimulation; innovation in the preservation or development of the artform; artist development; and community relevance.

A key issue for DCA is the level of commitment it wants to display in measuring quality and the corresponding resource implications of adequately funding peer reviews of artistic vibrancy or excellence. For example, bringing in national and international peers is an important part of a robust process, due to the inherent danger of a ‘closed’ artistic community within a particular state or country indulging in ‘grade inflation’ to support each other’s claims for ongoing funding. The cost implications of such an approach are also self-evident.

3.11 THE ARTS AND CREATIVE INDUSTRIES

Finally, we have worked with DCA to explore possible approaches to measuring the outputs and outcomes from the wider creative economy in Western Australia. There is burgeoning literature on the links between the creative or cultural industries9 and their relationship to publicly funded arts and culture. The list of creative industry segments is now becoming very familiar to everyone in the arts:

• Music and performing arts

• Film, television and radio

• Advertising and marketing

• Software, web and multimedia

9 The Work Foundation (2007) ‘Staying Ahead: the economic performance of the UK’s creative industries’ DCMS; NESTA’s work in the UK; Throsby, D (2007) ‘Modelling the Creative /Cultural Industries’ in Holden, J. (2007); Cunningham, S (2006) ‘What price a Creative Economy’, Platform Paper, No. 9, Australia Council

14


• Writing, publishing and print media

• Design and Visual arts

• Architecture

These businesses focus on linking creativity with commercial markets and use creative output within their value creation. They transform ideas into intellectual property from which they extract money through entrepreneurial marketing and sales processes. The arts contribute in both direct and indirect ways to the growth and success of the creative industries, through direct transfers of product, skills and ideas, and in creating the cultural ecologies in which creative industries thrive10.

All too often, treatments of the arts and creative industries focus on their differences rather than their similarities. Yet there are many common elements between the subsidised and unsubsidised arts, as summarised by Caves11:

• Considerable uncertainty about the likely demand for creative product, due to the fact that creative products are ‘experience goods’, where buyers lack information prior to consumption – and where the satisfaction derived is largely subjective and intangible

• The ways in which creative producers derive non-economic forms of satisfaction from creative activity – but each are reliant on basic business systems (e.g. accounting and marketing) to make their activities economically viable

10 Holden, J. (2007.7) ‘Publicly funded culture and the creative industries’ Demos

11 Richard Caves, Creative Industries, Harvard University Press, 2000

• The collective nature of creative production, requiring development and maintenance of diverse creative teams, often under severe time and budget pressures

• The wide array of creative possibilities available to the creator

These similarities and the burgeoning links between the arts and creative industries, underlined the importance of creating the value framework to actively consider these linkages and relationships. The intention being to ensure that any metrics developed within the value framework are capable of capturing these dynamics of innovation, growth and commercialisation linking the arts and creative industries.

15

4.0

4.1 DCA FUNDING PROGRAMS

This chapter will not examine DCA funding programs in a detailed manner. The reader should refer to separate reports on the funding process review and acquittal data review projects that are occurring in parallel with this project. The purpose of this chapter is to highlight the number and range of indicators for DCA funding programs and the degree to which these indicators support policy and feed back vital performance data to strategic resource allocation decisions.

DCA has a range of programs and each of these is assessed or acquitted in a different way. The principle of peer review is very important within the existing system, with panels convened to assess many of the applications.

The funded programs have been separated into four categories:

• Arts Grants

• Contemporary Music

• Designer Fashion

• Funded Organisations

The first three programs are assessed using a similar methodology incorporating panels, however the assessment process for Funded Organisations is slightly different and does not involve panel assessments. Each of these four programs is acquitted in separate ways using different acquittal information (and there are also further variations in the acquittal process for some minor programs).

The DCA funding system has a need for clarity on outputs and outcomes and a policy and measurement framework that is adequately precise on priorities and definitions.

The DCA policy framework does not lack specificity or sophistication, but through modelling and testing of the value framework it has been revealed that key elements may need to be distinguished more clearly and precisely. The modelling has already suggested a shortlist of possible priorities, outputs and outcomes which would aggregate all of the existing commitments into a more legible and easy to understand set of over-arching strategic objectives.

More work will also be required on refining the definitions, which will benefit from the rigor of the proposed measurement framework.

The sharper challenges concern the current operation of the measurement and evaluation system that underpins the current funding system, and how it can be improved to support the operation of the new PVMF.

4.2 APPLICATION, ASSESSMENT AND ACQUITTAL PROCESS

The funding and acquittal processes are currently under review and are being assessed in studies that are separate and independent of this project to develop a PVMF.

It was noted that the Application, Assessment and Acquittal processes are intricately linked to the execution of the PVMF. This is because all funding programs are derived from overarching policy determined by DCA and all processes to distribute those funds need to include reference to the higher-level policy. Finally the results from funding must be fed back to the higher order policy to inform government of the success of its programs and identify areas for improvement or change. This is where Decision rules can be adjusted over

4 CURRENT DCA MEASUREMENTS

16


time to ensure that funding programs continue to reflect government priorities.

4.3 METRIC INVENTORY

A study was completed of the existing assessment processes and all existing acquittal data to determine what information currently collected would be suitable to being used in a new PVMF. Of over 300 metrics which are collected in the acquittal process across the four funding programs, a total of 179 different metrics were identified which could possibly be linked to the outcomes of Quality, Reach, Value or Impact (see Figure 2). These measures were linked to a database with the intention that they could ultimately feed into the PVMF.

It was concluded that the majority of these measures are measuring outputs not outcomes and therefore provide little valuable insight into the PVMF. Whilst a small number of existing measures could be used to explain aspects of Reach, most of these measures are superfluous to the PVMF.

It was also noted that there is little relationship between the assessment process and the acquittal process. There is no consistency between the process used to assess programs and the process used to acquit programs.

It became obvious that in order to move forward with introducing a PVMF, the total number of metrics needed to be rationalised. The total metric count figure of 394 is obviously very striking. Both funder and funded will benefit from a reduction in this total metric count.

The amount of information sought from grant recipients through the acquittal process needs to be rationalised and standardised with the information sought at assessment

time so that there is a core data set common across all funding programs and common across assessment and acquittal processes. For example, the designer fashion program has developed 137 performance indicators for a single funding program.

Figure 2: Metrics by DCA

Metric Types Total

All metrics* 394

Public value metrics 149

Other Metrics 245

* Current acquittal process

More broadly, the four funding programs show different proportions of quantified metrics (see Figure 3) with key funded orgs at 100% (79 of 79) and arts grants/contemporary music at 20% quantified. 80% of metrics for these two programs are subjective and qualitative.

A detailed breakdown of metrics by funding program is provided in Appendix 2.

Figure 3: Metrics by Funding Program

Funding Programs Quantitative Metrics Total Metrics

Key funded organisations 79 79

Designer fashion 61 137

Arts grants 18 89

Contemporary music 19 89

Total 177 394

Essentially a single simple system is required applying universal definitions of Quality, Reach, Value and Impact across all the programs at both the assessment and acquittal stages. The system needs a closed loop to provide the opportunity for the system to learn and to record the learning events. It needs multiple points of reference from which to source

17


data which would be assessed against simple decision criteria. We explore all of these issues further in the next chapter.

A detailed breakdown of metrics by Quality, Reach, Impact and Value dimensions is provided in Appendix 3.

Because the entire process is under review, no recommendations have been made in terms of which aspects of the application or acquittal process need to be removed in order to simplify the whole process.

Further work will be needed to incorporate the PVMF into the new Funding and Acquittal processes.

4.4 FINANCIAL AND QUANTITATIVE HARMONISED REPORTING

In 2002 Commonwealth, State and Territory arts funding agencies developed the national arts organisations statistical data set as part of the harmonised reporting framework.

The harmonised reporting framework has established a set of agreed and unified financial and quantitative data items for all organisations co-funded by the Australia Council for the Arts and the respective state funding agencies.

The quantitative data items covered are primarily in relation to outputs from funded activities and fall into categories such as: number of performances, exhibitions or events; participant summary; attendance summary; and number of employees,

As these are harmonised data items across jurisdictions there is an ability to compare input and outputs to organisations, however the framework in itself does not provide an

overview of the outcomes delivered from this investment.

4.5 DCA KEY PERFORMANCE INDICATORS

As discussed in chapter 3.4, the DCA assesses its performance against creativity, sustainability, accessibility and preservation, criteria as discussed below.

4.5.1 A creative, sustainable and accessible culture and arts sector

This outcome has three effectiveness indicators, creativity, sustainability and accessibility.

Creativity is measured as the proportion of funding applicants who are satisfied with the key elements of the creative funding programs (biennial client satisfaction survey). Creativity is said to ‘depend on the cultural values, preferences and realities of residents and other stakeholders in a given community’. The resource allocation decisions are made in each of the funding programs by peer panels. In 2010/11 67% was achieved.

Sustainability is measured in two ways: 1) by the perceived value of the culture and arts to the WA community and 2) the proportion of triennially funded organisations within the culture and arts sector which are regarded as financially healthy. For 2010/11 these two measures were 77% and 12% respectively. The first measure provides an assessment of the extent to which the community values the culture and arts sector (measured through the Arts Monitor). The second provides an indication of the financial health of the triennially funded organisations that make up a significant proportion of WA’s subsidised culture and arts sector.

18


Accessibility is measured as the proportion of Western Australians that perceive culture and arts events to be accessible (gathered through the Arts Monitor). In 2010/11, 61% was reported.

The Key Efficiency Indicators for the DCA portfolio relate to cost per grant and cost per performance.

4.5.2 Western Australia’s natural, cultural and documentary collections are preserved, accessible and sustainable

The first Key Effectiveness Indicator for this outcome is the extent to which WA’s natural, cultural and documentary collections that require preservation are preserved (as a percentage). This provides an assessment of the extent to which the State’s art, museum, heritage and archives collections are preserved. It is explained that since preservation of the entire collection is not required, this measure only relates to that part of the collection that is determined by the organisations as requiring preservation. So the indicator measures the extent to which the portfolio’s collection preservation outcomes are met.

The second Key Effectiveness Indicator relates to accessibility and is assessed based on 1) the number of accesses to WA’s natural, cultural and documentary collections per capita; 2) percentage of clients satisfied with the services associated with accessing WA’s natural, cultural and documentary collections and 3) number of accesses (attendance figures at Museum, Art Gallery, State Library and State Records Office and accesses via internet user sessions and material exchanges between State Library and Local Government Public Libraries. In

2010-11 they were 2.063, 93% and 4,779,778 respectively.

The third Key Effectiveness Indicator relates to sustainability. This is the value of collection renewal content development, expansion and or maintaining the physical integrity of the state’s art, museum, heritage and archives collection as a proportion of collection value. In 2010-11 6% was reported.

The Key Efficiency Indicators for the second Government Outcome all relate to cost of service delivery/access for the Art Gallery, State Library, Museum and Government Record Keeping Services (i.e. relating the resource input to the services provided).

4.6 KPI REVIEW

The DCA’s KPIs are currently under review. The issues to be addressed by the review are:

1. The existing creativity measures do not accurately measure creativity

2. Key effectiveness indicator for Accessibility is based on community perceptions measured through the annual Arts Monitor survey but do not directly relate to DCA core processes, and therefore DCA is unable to directly influence the measures through changes in performance

3. The key efficiency indicator for Arts Industry Support is not an accurate reflection of efficiency and is based on historical calculation assumptions that may no longer be valid

4. There is limited, if any, use of current KPIs in the day-to-day management of the DCA

19

5.0

This chapter explores the possible improvements to the value framework and existing funding processes, which have emerged out of the modelling of the measurement framework and the testing of the framework and DCA’s funding process.

5.1 TOWARDS A SHARPER SYSTEM

The PVMF needs to be applied at two levels: at the policy making level (yielding robust and widely understood terms and priorities); and embedded in the funding application, assessment and acquittal processes (direct feedback to the policy level).

In practice, current measurement and evaluation attempts in the arts are usually undermined by:

• Policy that doesn’t clearly articulate what outcomes are desired and therefore how scarce resources (public and private) should be allocated to achieve them

• A lack of distinction between outputs and outcomes. It is important when evaluating and designing measurement and evaluation systems to be clear about the distinctions between outputs and outcomes. Outputs can be defined as direct, measurable results of activities undertaken (e.g. the number of people who attend a show, the amount of funding received). Outputs are normally the subject of the public audit process, which is a key compliance driver of public funding systems.

• Loose definitions that cloud the true meaning of desired outcomes – so politicians, policy makers, artists and

audience can say that they want the same outcomes but mean different things. This leads to confusion, unfulfilled expectations and claims of bad policy and poor implementation. For example, the use of terms like ‘quality’ and ‘excellence’ are beset by the negative consequences of inconsistent definitions. In public policy terms, it is in fact better to get a definition ‘wrong’ if we all agree on it, than to get it ‘right’ and be in disagreement.

• Ineffective implementation systems. Even if policy is clear and definitions are universal, if the processes by which outcomes are measured are highly variable, the system will break down. For example, under a normative scoring system, if different funding streams are using different scoring scales – then variation will occur.

Outcomes relate to the changes resulting from the activities that have produced the outputs. It is easier to measure the number of people attending the show than the impact on them as a result of their attendance. Outcomes can be both tangible and intangible. Tangible outcomes occur where:

• The output is the desired outcome

• The outcome results from a market process and has a measured price

• Some other technique of market value estimation can be applied to effectively measure the outcome

The temptation with any measurement system is to focus on what is easier to measure (outputs) rather than what it is important to measure (outcomes).

5 A NEW APPROACH – THE PUBLIC VALUE MEASUREMENT FRAMEWORK

20


5.2 PUBLIC STAKEHOLDER SESSIONS WITH FUNDED ARTS AND CULTURAL ORGANISATIONS

During late 2011 and early 2012, DCA and Pracsys hosted a number of stakeholder sessions, ranging from small discussion groups to a large public session with over fifty attendees, in which we tested out the rough outlines of the public value measurement framework. Some of the dialogue was focused on establishing a shared understanding of the emerging measurement framework and in addition participants made a wide range of observations on the framework.

A list of meetings and workshops conducted is provided in Appendix 4.

Overall there was significant support for the measurement framework. Participants displayed a sophisticated understanding of the potential benefits of the framework, whilst being keen to establish that it could effectively capture their artistic activities and practices in a sensitive, non-distorting manner. Participants also expressed clear expectations about what the measurement framework needs to be able to deliver, which included the following key points:

• That the measurement framework is capable of making the case for public investment in the arts much more strongly in Western Australia

• That it can effectively capture how public investment in the arts and culture is leveraged – both in monetary terms and in terms of who else gets the chance to practice / participate in cultural activity

beyond the funded artist / institution

• That it should help to make the funding system more transparent and accountable

• That it allows the arts ‘to better present the value we create currently, and the value we can create in the future’

• That is sensitive to value creation in the creative economy / digital spheres

• That it needs to be future proofed, in the sense that it takes account of new social media / digital based metrics and analytics

• That it better captures the cross-overs from individual consumption of arts and culture to active participation in productive / creative processes

• That is needs to capture important place-making and liveability outcomes (the vibrancy of a place and its attractiveness) associated with public investment in the arts

• That it can over time influence the way in which the Australian Bureau of Statistics measures participation, allowing for a better measurement of creation and consumption

Concerns and questions were also expressed about:

• The relationship between the measurement framework and future investment processes

• The potential over-metrication of decision making – ‘how will this measurement framework shape the funding decision making process?’

21


• The ability of the measurement framework to deal with different arts forms effectively

• The ability to measurement framework to capture changes in artistic quality / excellence over time

• Whether it will be able to more sensitively capture local community engagement and participation

• The potential for harmonisation between the measurement framework and the requirements of other funders

• How DCA will work with other agencies to capture some of the wider, induced benefits of public investment in arts and culture

• The costs and usability of any new scheme as part of DCA’s funding process

• How best to engage other public investors and stakeholders in developing the model, so that it could be applied in other related areas (sport; museums and heritage etc.)

• How to build national credibility for the measurement framework, with observations that this might need to be a separate stage, engaging with The Australia Council, other State Governments and relevant research councils

In subsequent development of the value framework, the concerns and challenges were noted and addressed.

A detailed account of the stakeholder session is contained in Appendix 5.

5.3 PEER REVIEW

During the week of the public sessions in Perth in October 2011, the Pracsys team discussed with DCA what the most appropriate form of peer review would be for the emerging PVMF. It was agreed that John Knell focus his efforts on Arts Council England as practical, funder based reflections on the model are of much higher use and also offer a strong reputational endorsement of the model if key ACE staff are supportive of the PVMF.

There has also been detailed input from two senior Executives inside ACE, namely Andrea Stark, who is on the top team of ACE, and Catherine Bunting, who is Director of Research at ACE, and who ran their public value inquiry three years ago. In face-to-face meetings with Andrea and Catherine, we discussed benchmarking, the public value logic model and the array of ‘quality’ and ‘reach’ metrics.

Their key responses are summarised below:

Firstly, both felt that the model was coherent and represents an advance on anything that is being attempted elsewhere. Andrea Stark thought the model was ‘clear and exciting, in that it moves the sector beyond where we are.’ Catherine Bunting commented that:

‘I’m particularly excited by the model because it is holistic – it offers a complete map of the process of public funding of culture from decision-making through to the assessment and articulation of value. In this sense the DCA is breaking new ground and I’m not aware of any other funding body internationally that has made such progress in develop a strategy for maximising the public value of culture. The proposed framework is simple without being reductionist and should

22


lead to greater transparency, understanding and trust between the DCA, its clients and other stakeholders.’

Other general points they noted about the PVMF model include the following:

• A logic framework is being applied to an area where there’s been very little rigorous logic applied

• That the framework lays the foundation for consistent measurement, comparison and aggregation across funded organisations in areas such as artistic excellence and innovation and audience size, diversity and experience

• That by closing the loop back to decision-making, the ‘framework encourages the DCA to take responsibility for the value it creates rather than treating value as something to be asserted as part of an advocacy campaign’

Beyond these general points about the model, Andrea and Catherine identified some operational key implementation issues:

• The challenge of identifying standard methods and data collection systems to enable organisations to report on quality and reach in a consistent way

• Developing a full taxonomy of social and economic impacts and providing methodological support to enable organisations to measure their progress against the impacts most relevant to them. It may be more cost-effective for DCA to undertake impact assessments on behalf of all, or a subset, of its portfolio

• Deciding whether and how value should be measured. If the DCA wishes to directly compare the value of the impacts

achieved by different organisations, the standard approach in other sectors is to use economic valuation techniques (such as contingent valuation) to assign a monetary value to different types of impact. Experience suggest that this is likely to be a highly complex and expensive process and perhaps not something that DCA would want to undertake for all of its portfolio or on a frequent basis

• The challenges around operationalising the model. For DCA to be successful in implementing this model they will need to build a strong collective sensibility across the whole portfolio of funded organisations. As Andrea commented:

‘Whilst it is important the model is sensible and comprehensive, which this is, DCA will also need to be able to shift mindsets around all of this.’

• One option here would be for DCA to explore with the sector how they can encourage them to directly adopt the PVMF as a self-evaluation tool that they will use to shape their business planning activities. The great benefit of this approach is that the PVMF framework would not feel like a ‘top down’ mandated outcome set, but rather a set of outcomes that are supported by the sector and that they are actively using to shape their forward planning

• The need for DCA to strike the right balance between a technocratic, numbers driven evaluation system and the need for expert judgment by art form professionals, balanced with whole portfolio decisions (geographic issues; diversity issues; cultural entitlement issues). As Bunting commented:

23


‘We attach great importance to the judgement of individual experts and to extensive periods of debate, negotiation and consensus-building both internally and externally. Evidence plays a part, and there are certainly a number of elements of the PVMF that we would find very helpful, but to adopt it in full may be too much at odds with our identity as an organisation.’

• The importance of DCA developing a very clear set of prioritised objectives to underpin the use of the PVMF and an equally clear view of how it wants to operate as an organisation and funder, in terms of its evidence based approach and decision making culture. As Catherine Bunting commented:

‘Perhaps the biggest challenge now for the leadership of the DCA is to determine how well the proposed framework matches the DCA’s vision of the kind of organisation it wants to be.’

• Thinking through how best to make the case for the PVMF, which ‘needs to look simple and comprehensive enough to convince Ministers and officials that it’s a turn key solution for telling the return on investment story.’

• The PVMF also needs to provide a very clear answer to what issue is it seeking to address. In other words, what is the ROI on the PVMF itself? What are the benefits that will accrue from getting it right in the next five years? Some of the key benefits noted by these Arts Council England executives included:

1. To align the self evaluation of funded arts organisations more closely with DCA investment priorities

2. To convincingly tell the ‘return on investment story’

3. To allow DCA to tailor its strategy and evaluation to speak powerfully to the wider priorities of the Government

4. To unlock value in other state departments, by encouraging joint approaches to investment and evaluation, particularly those departments where arts and culture do not currently have strong relationships

In summary, the feedback from Arts Council England was positive, confirming:

• That the model is innovative and coherent

• That it should allow DCA to capture a rich and diverse range of impacts achieved by individual organisations and the portfolio as a whole

• That it would represent a leading edge attempt to provide a strong evidence base for DCA’s decisions about support for culture and the arts in Western Australia

• That its successful implementation requires DCA to generate strong support for the framework amongst funded organisations and other funders and investors

• That DCA needs to think carefully about the implications of the framework for its current assessment and evaluation process, and that it will need to find the right balance between a technocratic, numbers driven evaluation system and the need for expert judgment by art form professionals, balanced with whole portfolio decisions

24


Both Andrea and Catherine signalled their willingness to have further discussions with DCA as they develop the model and move into the implementation phase.

The early stage testing of the model has been very useful in refining the suggested dimensions of quality, reach, and impact.

The artistic practitioners who have been involved in the testing process engaged very openly and productively with the potential metrics, and will continue to add value to the model in further testing rounds. The process has underlined the potential power of a PVMF that is jointly shaped and authored by both funder and funded, which reflects artistic intention and aspiration as well as funder ambition and responsibility.

There has been some disquiet among practitioners about the suitability of standardised rating scales allied to the acknowledgement that the movement towards a more consistent and meaningful assessment and measurement process will require a standardised reporting framework.

Participants in the testing also recognise that designing the funding system so that it provides the opportunity for both the funding applicant and the peer assessor to review their pre-output assessment after the event would greatly improve the year on year assessments and operation of the system.

5.4 OPPORTUNITIES FOR CO-INVESTMENT

Although the impetus for establishing an evidence base for instrumental, intrinsic and institutional arguments in favour of

arts and cultural funding comes from the DCA in this instance, the outcome will have application for any funding source. A non-DCA funder’s decision to fund (or not to fund) particular activities will be based on subjective assessments and ‘patronage’ style decision mechanisms, largely because the arts and cultural sector has never developed a comprehensive measurement system to argue its case at any level. Implementing the proposed PVMF will provide an evidence base and a platform for value-based discussions with other government agencies (e.g. health, education, community development) based on standards of proof common to those portfolios (but not yet to DCA). This will enable stronger and longer-term strategic engagement with those (and other) portfolios and inevitably result in a more diverse funding mix for the arts and cultural activities across the public sector.

There is also a clear opportunity to use the PVMF to influence federal funding arrangements by establishing a superior evidence base on which to lobby increases in future funding contributions from the Commonwealth.

With regard to private funding sources, only a few well-led and entrepreneurial arts and cultural organisations have been able to secure significant private sector contributions in support of their capital and operating requirements. This is partly due to the immature nature of the corporate and private philanthropic giving culture, but this is not the only reason. The general inability of subsidised arts organisations to understand the felt-needs of corporate and private partners and to design outcomes to meet those needs has also greatly contributed to the low diversity of the funding mix. A result has been an over-reliance on public funding, fostering a mendicant attitude

25


towards public funders, which has been to the detriment of many capable arts and cultural organisations.

By embracing public value metrics, organisations will be able to argue their cases to the private sector with a new confidence and awareness of what they have to offer. More importantly, they will be able to align what they have to offer with the needs of potential partners for mutually beneficial outcomes.

5.5 PRACTICAL TESTING OF THE VALUE FRAMEWORK

The key public value metrics specified by DCA are quality, reach, impact and value.

The practical testing with individual arts organisations, including the Western Australian Symphony Orchestra and West Australian Opera Companies, explored the definitions within the public value measurement framework. It is encouraging to be able to report that the funded organisations have been positive about the PVMF, recognising its potential value both to them and the broader sector if it allows them and others to make a stronger case for arts funding and increases public understanding of the value of arts and cultural activity.

The practical testing sessions have greatly enhanced the understanding of how each metric may be defined, suggesting new dimensions to quality, reach, impact and value (see Figure 4). We have summarised the resulting set of the new metrics in forms that could be used at the assessment stage (Form A) and at the acquittal stage (Form B), combining the proposed metrics with standardised rating scales that would be used for self, peer and public assessment (see Appendix 6).

5.6 A PUBLIC VALUE MEASUREMENT FRAMEWORK MODEL

The notion of public value from funded arts and cultural activity derives from:

• the quantified outputs in the form of attendances, box office receipts, sponsorship etc.

• the intangible impacts of the quality and reach outcomes

• long term public value infrastructure that enhances the sectors ability to generate repeated impacts

When set against the amount of investment required to generate the funded outcome, the sum of these three types of impact can be used to derive a value estimate. If this process is repeated across all funded activities, it is possible to compare values for different outcomes across art form, funding program etc. This is the key to effective resource allocations and developing an optimum funding mix. The system also enables the development of quantified decisions rules and alignment of artistic and cultural outcomes against policy objectives.

The following definitions have been used in testing the public value measurement framework.

26


5.6.1 Quality

DCA’s Creating Value Policy states that:

‘Quality is seen as the measure of the creative process and product and will include the distinctive, innovative and significant elements of the creative experience. Quality will be measured through a combination of self-assessment, peer assessment, audience and public response.’

What the testing round has revealed is that arts practitioners think that quality is comprised of the following key elements:

Creativity:

Which in turn is made up of:

• Inquisitiveness – the extent to which the funded activity promotes curiosity in artist and audience

• Imagination – the extent to which the funded activity explores new possibilities and views

• Originality – the extent to which the funded activity breaks new ground (modes of practice and content)

Risk – the extent to which the artist is fearless and negotiates new artistic approaches

Figure 4: Public Value Measurement Framework Process Diagram

Source: Pracsys 2012

27


Rigour – the extent to which the funded activity has undergone thorough research and development

Currency – the timeliness of the creative idea in relation to contemporary events

Authenticity – the extent to which the funded activity respects cultural tradition or is uniquely Western Australian

Innovation – the extent to which the funded activity demonstrates an ability to realise creative ideas into real world outcomes

Excellence – the funded activity is widely regarded as best of its type in the world

5.6.2 Reach

DCA’s ‘Creating Value’ Policy states that:

‘Reach is seen as a measure of the access to and participation in arts and cultural activities. It will measure the breadth and depth of engagement, through attendance and participation data and alongside audience and public satisfaction with their level of engagement’.

What the testing round has revealed is that arts practitioners think that Reach is comprised of the following key elements:

Audience

Measured through:

• Number – the number of people in the target communities of interest who directly engage with the funded activity

• Diversity – the extent to which the funded activity engages a broad cross section of society

• Connection – The quality of the connection of the funded activity with communities of interest

Practice - The extent to which the work connects with communities of practice

Leverage – The ability to attract investment from a range of non-DCA sources

Platform – The capacity of the work to have long-term influence and importance to communities of interest and practice

5.6.3 Impact

DCA’s ‘Creating Value’ Policy states that:

‘Impact is seen as a measure of social, cultural and economic impact of arts and cultural activities, and will be measured through the quantitative and qualitative review of outcomes sought from engagement’

The engagement with artistic practitioners, and indeed the testing with DCA staff, has revealed that a dimension which has not been formally integrated within practical or academic treatments of cultural/public value measurement, is that all of the dimensions of impact have a common theme of transformation – that is the extent to which the funded activity changes how artists (and their communities of interest) and audiences (and their communities of practice) think and feel about their world. Whilst this dimension poses significant challenges in terms of standardised scale measurement, we have included it in the draft Form A and Form B given the strong feedback from artistic practitioners, and so that we can explore the ways in which this metric might be framed in further testing rounds.

28


The other interesting element to emerge out of the testing phase is that the arts practitioners regard the impact category as having both a ‘narrow’ and ‘wide’ dimension, and this implies distinctively different roles for themselves and for DCA as the public funder. The ‘narrow’ impacts, the definable and direct cultural (intrinsic and institutional) and economic impact arising out of their activities, are the responsibility of both DCA and the funded sector. DCA must sponsor and support the individual arts organisations to report against any new measurement framework and work with them to both refine the measures over time, but also to find new ways of generating relevant data, making full use of social media as a reporting technology linking audiences and organisations together.

5.6.4 Value

DCA’s ‘Creating Value’ Policy states that value is seen:

‘As a measure of the economic value of arts and culture and the appreciation of arts and culture in the Western Australian community. Value will be measured through both the return on investment and the value of culture and the arts through community surveys.’

What the testing round has confirmed is that arts practitioners fully understand that value has a layered and complex meaning. First, value has strict narrow economic meaning, as in the Creating Value definition, but it also has intrinsic and institutional value (ref Holden and Throsby). Added to this, it has ‘externality’ value in that it creates skills and experiences that are vital to the so-called creative industries (advertising, gaming, film, television, etc). The research also found that value has a broad

community meaning (the ‘wider impact’ approach) that can be measured using contingent valuation and similar techniques. All of these elements should be considered together to gain a full picture of the true meaning of value.

5.7 HOW DO THE ELEMENTS OF PUBLIC VALUE FIT TOGETHER?

These elements should be considered sequentially, rather than all together as implied in the Creating Value framework. Quality and reach are directly related to the inherent attributes of the outputs produced. In a narrow sense, the impact is a product of the quality and the reach of the output. If the output is of very high quality but reaches few members of its target community of interest, it will have a lower impact than if its reach was greater. If it has low quality but an extensive reach, similarly, it can be regarded as having a lower impact than if quality was higher. If quality and reach are both high, impact is high. From the wider value perspective, value should be measured from a whole-of-sector view, based on contingent value and other studies commissioned and disseminated by DCA.

5.8 DEFINING PROCESS REFORM

So how might the current system be improved, in light of our modelling of the PVMF and our testing of its implementation? Firstly, work needs to be done to remove subjectivity around the definitions. This would be helped by adopting a standardised semi-quantitative scale for each metric element as demonstrated by our suggested Form A and Form B (see Appendix 6 for prototype). The artistic practitioners who took part in the testing

29


expressed some concerns about using such standardised scales. The harsh truth is that some sort of standardised reporting framework is a necessary discipline to achieve consistency in the assessment process.

Under the proposed system, each measured dimension uses the same quantified rating scale, which removes some of the spurious variation and subjectivity associated with using open-ended questions. Another advantage is that rating scales generate quantitative data that builds a statistically significant sample as the number of observations grow over time. In other words, although each self or peer assessment is necessarily subjective, the sum of all observations builds a data set that becomes objective when the sample size is large enough.

If these improvements to both the definitions of terms and the method of data collection can be secured in the future, the system will be able to fulfill its real purpose – to measure differences or similarities between what the funding applicant thinks of their work/activity and what their peers think of their work. Moreover, this

standardisation in the application process will enable comparisons between the activities of the same artist/art-form over time – and even comparisons between funding programs and activities from different art forms. If used more widely, it would also be useful as a benchmarking tool with funding programs in other jurisdictions (e.g. other states across Australia).

Another significant weakness of the existing application-assessment system is that it only gathers data about what the applicant and the relevant peers think about the funded activity before it happens. There is no systematic checking of what the applicant and peers think after the output is produced. As with many risky activities – the output is often very different from what was expected. In other words, the system funds activities based on expectations, and then is incapable of checking to see whether those expectations were met. The system is therefore incapable of learning.

So what does a system with learning embedded within it look like? A learning system has to provide the opportunity for both the funding applicant and the peer assessor to review their pre-output assessment after the event. The analysis of variance on their respective assessments will give each the opportunity to learn how the output was different from their expectation – equipping them to make a better judgement next time.

A third and vital element in the post-output assessment is the view of the public. This is a public value measurement framework after all – so the views of the public about what they value is essential. Under the current system the view of the public is sought irregularly and in a rather ad hoc manner. The exception is

Figure 5: Self and Peer Assessment


30


the annual Arts Monitor run by DCA – but this is very general in nature rather than specific about public response to specific funded activities. The surveys that are undertaken are also manual, paper based and expensive to administer. Some of the better-resourced companies also conduct their own market research, but they of course have proprietary ownership of this intelligence, so it seldom if ever available for wider use.

By gathering data from two sources before and three after the funded output, DCA would end up with seven observations on each funded activity. In the case of major organisations and annually funded artists, the system could run multiple times per year. So the number of observations would grow quickly once the system was introduced, providing a rich source of public value data in a learning process.

But surveys are expensive and beyond the reach of individuals and many funded organisations both logistically and financially. The solution is to run an integrated online system using the same assessment criteria for application and acquittal regardless of who is doing the assessing. DCA is well advanced in its planning for an online grants system, which could be extended to allow web access for funded organisations and individuals to conduct their post-output assessments. This initiative could also be extended to a mobile application (e.g. an iPhone app) for the public to register their feedback about the quality and impact of the funded output when they ‘consume’ it. Under this system, data collection is seamless, uniform, inexpensive and continuous – moving away from the infrequent, cumbersome and expensive manual and largely ad hoc surveys currently undertaken.

31

6.0

6.1 POLICY AND MEASUREMENT PRINCIPLES

An effective measurement framework needs to integrate policy, the measurement definitions, and the valuation and funding process (as outlined in Figure 6). In other words, the measurement framework has to have distinct links back to policy and funding decisions so that the framework is a dynamic informer of an evolving public value contribution.

Figure 6 captures the key building blocks to creating the measurement framework including a feedback loop, policy, measurement definitions and process. The policy objectives and outcomes need to be defined with enough clarity and precision to be able to be matched with appropriate measurement criteria. For example, if achieving artistic quality were a funding priority, it would be expected that the dimensions of quality have been modelled in ways that are understandable and measureable, but also credible to the funder and the artistic community.

The metrics that constitute the public value measurement framework need to meet some clear criteria, in that they need to pass a rigor and credibility test. In developing the measures within the framework, there have been testing sessions with artists and cultural leaders to try to generate metrics that effectively capture the practice and intentions as well as satisfying the desire of the funder to better account for the outcomes and value produced. If the needs of only one of these groups are met, the measurement system will fail.

As the middle box also implies, any robust measurement system needs to have a number of essential characteristics, as follows:

Simplicity: Due to the arts sector displaying a complex array of variations across art forms and outcomes, the measurement system needs to be simple enough for the entire sector to use it. It needs to show links back to the policy and display uniformity so that all measures are consistent no matter where they are applied.

Standardisation: Any measurement framework needs to be standardised so that the same set of parameters are being used, resulting in a common understanding of what to assess. When the information gathered across the whole sector is standardised, the various programs or art forms can be usefully compared and are able to be considered for roles in contribution to the achievement of stated public policy objectives.

Documented Decision Rules: Integrity of a measurement framework is ensured when decision rules are in place to provide a framework and accountability for decision making. Decision rules need to be created to provide a feedback mechanism to the DCA policy.

6 STRATEGIC ALIGNMENT OF METRICS TO POLICY, GOALS AND DECISION RULES

Figure 6: Creating an operable measurement framework


32


Evidence base: An effective measurement framework must be evidence based so that the integrity of decisions made can be ensured. Over time, an effective measurement framework generates a record of the evidence base. This record reinforces the capacity of the system to ‘learn’.

The alignment of the policy and measurement definitions also need to produce an evaluation, funding and acquittal process that is seen to produce robust, credible and transparent outcomes. The outcomes from the funding and evaluation process need to feed back into the policy decisions and future resource allocation rounds.

There would be little point in developing a new measurement framework unless it has tangible impacts on the way arts organisations plan and report their activity and on subsequent allocations of public funding.

The feedback loop, encompassing policy, measurement definitions and process, ensures that the whole measurement framework becomes a learning system. A robust measurement system must include the capacity to learn from itself. This means feedback loops must be incorporated so that the performance of organisations and projects within the funding process is measured and the results from the measurement inform ongoing decisions about further funding, policy and organisations or projects receiving funding within the system. The closed loop provides the evidence upon which future decisions can be made and provides the record of learning events, which reinforces the integrity of the system.

6.2 DISTINGUISHING AND ALIGNING POLICY CHOICES AND MEASUREMENT DECISIONS

In developing the measurement framework to integrate with DCA’s policy frameworks and these policy and measurement principles, we worked with DCA to refine their existing policy and output area priorities.

DCA already has a developed set of policy and strategic objectives.12 These objectives include two underpinning public value principles, being creativity and engagement. Each of these principles has a derived number of supporting goals (see Figure 7).

Figure 7: DCA’s priority public value principles

Creativity:

• A distinctive and enriching body of work unique to Western Australia

• Western Australia is a hub for critical debate, research and artistic practice

• Local, national and international recognition of Western Australian culture an arts significance

Engagement:

• Participation in Western Australian culture and arts activities

• Access to culture and arts activities for all Western Australians

• The value of culture and arts to all Western Australians

DCA also specifies key words for measurement in relation to public value delivery, namely ‘enriched’, ‘unique’ and ‘transforming’.

12 Department of Culture and the Arts, ‘Strategic Plan, 2010-2014’, and ‘Creating Value: An Arts and Culture Sector Policy Framework, 2010-2014’

33


It is also stated that to measure the delivery of public value through the principles of ‘creativity’ and ‘engagement’, the Department measures key drivers of public value as quality, impact, reach and value. These four measures are aligned against the DCA visions and the creativity and engagement principles.

• Quality = Unique, Innovative, Distinctive

• Impact = Enriching, Transforming

• Reach = Participation, Access

• Value = Recognition, Significance, Value for Money

The final element of DCA’s policy framework is the four key outcome areas. These are creative people, creative communities, creative economies and creative environments. Under each of these outcome areas, DCA lists some priority outcomes.

To develop the measurement framework, a shared understanding of what is meant by the terms, quality, reach, impact and value was generated in order to develop agreed definitions for both the outcome areas and policy drivers/goals. This generated a wide range of commentary back to DCA on where there is potential confusion or over-lap between its stated policy drivers/goals.

The project team coordinated a series of facilitated meetings with DCA staff in which the key outcome areas and drivers were addressed, identifying the needs/outcomes of the measurement framework.

The priority outputs and activities that DCA expect to see under each of the four key outcome areas, creative people, creative communities, creative economies and creative

environments, were reviewed in conjunction with exploration of the quality, reach and impact of each outcome.

This led to a new set of output area priorities as follows:

Creative People Output Area Priorities

• Support and strengthen the skills of the arts workforce, with a particular emphasis on digital skills

• Generate popular and public policy support for the arts

• Ensure more people experience bold and innovative art

Creative Communities Output Area Priorities

• Generate deeper and richer levels of engagement with the communities we serve

• Encourage communities to create and share their stories

• Build cross sector partnerships to broaden the creation and delivery of culture and arts experiences

Creative Economies Output Area Priorities

• Encourage economic sustainability

• Support real and virtual clusters and cross sector collaborations

Creative Environments Output Area Priorities

• Support cultural hubs that scale cultural and commercial activity, attracting talent, audiences and a diversity of expertise

34


• Better understand and support the key assets and networks driving the creation, consumption, distribution and conversation of cultural outputs

• Support infrastructure that will enable artists and communities to create a more vibrant public realm in WA

These priorities create the output frame against which we sense checked the emerging metric set of the public value measurement framework. In other words, the measurement framework has therefore been modelled in a way that would allow DCA to capture these prioritised outcomes through its investments.

Possible innovations in the modes of measurement and types of data collected were discussed in particular, the use of social media as evidence of reach and engagement among an audience. To include these sources of data within the measurement framework, the reporting system needs to make this possible.

The measurement process could be enhanced if funded arts organisations agreed to provide their main public funders with passwords to their Google analytics data. This could generate an overall sense of ‘reach’ in the sector through digital channels with the ability to provide feedback on the relationship between digital contact and participation with audiences.

Another important element for the measurement model was to establish how far the measurement frameworks design captured direct outcomes from activities (in terms of cultural production, distribution and participation). Other indirect or induced outcomes (wider educational attainment or wellbeing outcomes) would be captured by other measurement interventions outside of the measurement framework.

6.3 THE ONGOING DEVELOPMENT OF DCA POLICY AND THE PVMF

The key supporting change that will be required to implement the PVMF is to secure strong strategic alignment of metrics to policy, goals and decisions rules. For DCA this will mean that there is strong alignment between its strategic policies and its funded portfolio of activities, with metrics in the PVMF functioning as the link between them. This will ensure that the value framework is used as a practical decision tool in shaping the strategic direction of DCA.

Therefore, DCA’s policy will need to be restated and tweaked using the same language and terms that are finalised within the PVMF.

The process of allocating funds through defined decision rules will also need to be integrated in exactly the same way with the PVMF terms and definitions. This is what will create a coherent system, with the PVMF directly linking policy, decision-making, funding allocation and acquittal processes.

What will then drive the whole funding process are the strategic decisions that DCA will continue to make about the portfolio balance it wants to achieve through its investments. DCA will be seeking to weight different elements of the PVMF to reflect its overall strategic priorities. Those weightings will segment the available public funds across different funding programmes / categories of client. The weightings will also shift the portfolio balance produced by the individual funding programmes.

35


Currently, there is a misalignment between DCA’s stated policy goals and what is actually funded through its programmes. Our recommendations will create a public funding system that has long-term credibility, in which DCA’s policy intentions will be fully reflected in weightings, decisions rules, and future allocations of resources.

What then is implied in practical terms by the integration of measurement definitions with policy and process? Looking beyond DCA for a moment, some public funders of the arts deploy a two stage assessment process in terms of the their overall funding decisions for the mainstream funding of their revenue clients (i.e. not specific programme or project funding). Stage One normally involves an assessment of individual applications judged against the stated policy priorities of the funder. Decisions to fund, or not fund, applicant organisations are made at this stage13.

Stage Two normally involves the assessment of how the resulting whole portfolio of funded organisations is contributing to the funder’s stated funding priorities, in terms of the desired balance of outcomes. This stage two process is therefore vital in shaping the final portfolio, confirming the level of investments across the revenue client portfolio that are required to best deliver the priorities of the public funder.

If we take our proposed measurement framework and consider a Stage One process, DCA would provide clear priorities to the applicants applying for funding about the relative importance of different measures within the framework. In other words, DCA’s clearly stated priority objectives would lead

13 See the process adopted by Arts Council England in their last funding round (2011)

them to give different weights to the different metrics within the measurement framework, which would be reflected in the resulting dashboard of measures for each applicant

For example, one may expect DCA to place a higher relative importance on some aspects of quality (creativity, innovation) than on some aspects of reach (monies raised from philanthropic income). The weightings used within the measurement system would reflect those priorities.

Assuming the weighting process successfully captures DCA’s strategic intentions as a funder, the measurement system would provide compelling evidence during this Stage One about which applicants the DCA should invest in and which they might reject. This Stage One phase would produce a shortlisted investment portfolio of funded organisations.

DCA might decide that this is where the influence of the policy weightings and measurement system end. During the Stage Two process, they might seek to bring in a range of additional criteria (geographic issues, diversity issues, cultural entitlement issues) to shape the final portfolio.

This would severely weaken the integrity of the overall measurement and funding process decision (this would effectively short-circuit the feedback loop required, linking policy with the measurement definitions and the decision making process). The logic of our approach is that the policy weightings applied to the measurement framework in Stage One, would be carried through to the Stage Two portfolio decisions. Nothing would prohibit DCA from augmenting their decision rules with some of the additional criteria such as those

36


mentioned above. But again, the integrity of the measurement system and the decision rules would be much greater if any such criteria (geography, diversity) have already been adequately captured within the measurement framework, which would then be supple enough to provide a powerful evidence base to support both Stage One and Stage Two assessments.

The pejorative and defensive response to this from seasoned art policy makers might be that this implies a technocratic, numbers driven evaluation system, which will be having a decisive influence on the shape of the funded portfolio. But this is to misunderstand what is being attempted here.

First, the resulting portfolio will reflect the clarity of the funder’s original strategic objectives and their integration with the measurement framework through the appropriate weighting of different metrics within the framework. The measurement system is policy neutral. It is the weightings used to drive the system that need to reflect funder intention. Clearly specified policy priorities, which should reflect the intentions of the sector as well as the funder, should produce supported and credible outcomes.

Second, the measurement framework will produce a greatly enhanced and improving evidence base upon which DCA can base its funding decisions and which others can also use to judge the ROI and value created. The commitment to using the framework doesn’t preclude DCA from using a wide range of expert practitioner input during the various stages of the assessment process, looking beyond the metrics and dashboard outcomes for individual funded applications, to both

check the veracity of the emerging outcomes for individual applications and to add an important layer of expert interpretation to aid DCA’s final decision making process.

37

7.0

We agreed with DCA that the final version of the model (as presented in this report), including a detailed discussion of some of the key metrics, could now be usefully tested outside WA, with Australia Council, other State Arts Departments, and leading academics in Australia.

This chapter reports on that testing process and the feedback we received on the model. All of the sessions involved a formal presentation from John Knell and Michael Chappell, and then a facilitated discussion of the model exploring the rationale, value and implementation of the model.

Australia Council Feedback

Nick Herd, Director of Research, chaired our session with invited colleagues from across the Australia Council. The key elements of their feedback are as follows:

• They acknowledge that this is interesting and valuable work

• They accepted that for the arts sector and public funders to work together on standardised common definitions and measures is a necessary and valuable task

• The room was split on practical aspects of implementing the system – in particular there was a concern about how would individual artists translate their artistic process through a standardised set of quality, reach, impact and value measures.

• In response we took them through how the metric framework already reflects the inputs of individual creative practitioners (for example the influence of the gaming community in WA on the platform measure)

• Frank Panucci (Director, Strategic Development Community Partnerships) made a very important observation that as you move from the ‘impact’ part of the logic model (which captures the ‘narrower’ quality x reach dimensions of value) to the wider induced benefits (education, place making etc.) – there is an important explanatory step that the current model doesn’t stress – which is that the more confident sector expression of impact (quality x reach) should then provoke a new ‘terms of trade’ discussion with other investors (public and private) about how to define and measure value in key value creating categories such as education, health etc. Frank felt this would be a powerful impact of the model as it is implemented.

• As we moved toward a close, a couple of the attendees expressed the view that they thought the arts sector would not embrace this approach

• We of course responded with a full account of the testing process and the positive reaction of the arts and cultural sector in WA thus far to the measurement framework

• In response, they urged us to make public in some form an account of the testing and stakeholder engagement process, and that if this were done quickly it would help build support for the model.

Arts Queensland feedback

Lorne Keast, the Director of Policy, Planning and Performance, and Shane Rowlands, Deputy Director, hosted the session with a wide range of staff from Arts Queensland.

7 TESTING THE VALUE FRAMEWORK – PEER REVIEWS AND FEEDBACK

38


The key elements of their feedback were as follows:

• They offered strong support for the rationale underpinning the PVMF logic model, and for its wider value to an arts funding agency.

• They were excited about the fact that for the arts sector the PVMF is an equitable and accountable system that allows everyone in the arts ecology (both individual artists and larger organisations) to tell a better value story

• They liked ‘the choral symphony’ dimensions – i.e. it should enable all investors, current and potential, and the arts sector itself, to begin ‘singing the same value song’

• As with other sessions – they quickly understood the power of this model to engage other investors and partners, public and private

• They expressed a clear understanding that engaging the funded sector in this type of exercise – particularly the co-production of shared definitions / metrics – is both necessary and valuable

• They identified another benefit of the model – which is that it will help move the debate away from a narrow focus on allocation, to a better conversation about efficiency and effectiveness

• The greater majority of the discussion involved Arts Queensland staff assessing where they are as a department in relation to the model – in terms of their application process and data management reforms.

• They offered warm support to WA for beginning this deeper exploration of how to capture the value of the arts sector, and expressed their willingness to input into the model as it is developed.

Arts New South Wales feedback

We met with a large array of staff from Arts New South Wales in a meeting hosted by their Director, Mary Darwell. The key points raised by Arts New South Wales were as follows:

• They think that the PVMF is a sophisticated model that offers a very good way of thinking about how the value of the arts and cultural sector can be better measured.

• They believe that other investors, both public (local authorities) and private, would welcome this model and be keen to support / develop it

• They were also able to make a quick leap from understanding the core value model to its application within other programmes, and shared delivery agendas (for example disability services – where they are looking to establish a public value based conversation between arts and disability services)

• They displayed a clear understanding of the importance of the wider public value infrastructure aspects of the model – and the delicate cost / benefit issues around investing more public money in capturing wider public value.

• They quickly understood the distinction between return on public investment and return on public value – as their policy has a strong emphasis on the wider value

39


creating elements of their interventions, in terms of wider creative economy and community benefit

• Like other sessions, they wanted to explore in detail the alignment of, and inter-relationship between, the output and outcome measures – and they clearly saw the potential for simplification and rationalisation, and were attracted to that possibility

Academic feedback

We met with two of Australia’s leading academics in the cultural policy sphere, Professor David Throsby of Macquarie University and with Professor Stuart Cunningham, Director of the Centre for Creative Industries and Innovation (CII) at Queensland University of Technology (QUT).

Professor Throsby is regarded as Australia’s pre-eminent cultural economist. His main comments were as follows.

• Firstly, that this is a valuable and leading edge piece of work that is filling a gap. David noted that up until now the debate about the value of cultural investment has stayed at a largely theoretical level, rather than being actively modelled from the explicit perspective of a public funder.

• He also noted that whilst it is obvious that a key foundation of our approach – namely to co-produce the measurement framework with the arts sector – is vital to any successful measurement framework – it is striking that this has not been attempted before. He is fully supportive of this approach.

• He believes that this is a good time for this model to be more widely discussed across Australia. He pointed to the publication of the National Cultural Policy in May as an opportunity to drive debate. More broadly, David mentioned that he has sat on a range of committees looking at the public value propositions of the National Museum, Gallery and Library, noting that the Federal Treasury not only has an increasingly open mind about public value propositions – but moreover would like to see more of them coming forward from the sector.

• He confirmed that he is very happy to keep inputting into the future development of the model.

Professor Stuart Cunningham is Director of CII. The centre draws on contributions from across the humanities, creative arts and social sciences to help build a more dynamic and inclusive innovation system in Australia, and is a recognised global authority on the creative industries and wider cultural policy.

Professor Cunningham met us both individually, and chaired a larger session with invited faculty members from across QUT. In terms of his one to one feedback, Stuart’s main observations were:

• That the proposed PVMF model is ‘rigorous, new, and hasn’t been attempted before.’

• That the model should be warmly welcomed by a wide range of public funders, and has potential applications outside of the arts sector

40


• That he strongly supports the project, and is keen to work with us as the project develops

The wider discussion with faculty members from across QUT offered a similar endorsement of the rationale and value of the model, and also touched on the implementation challenges associated with the model, and in particular QUT’s experience of developing a range of survey instruments that might be applicable to supporting the peer and public assessment elements of the PVMF.

41

8.0

8.1 RECOMMENDATIONS

The PVMF, and fashioning a new funding and evaluation system, will require DCA to produce its own project plan and consider available or additional resources for the implementation of the following recommendations.

1. Classify the core component definitions of the adopted public value model. This will involve defining more precisely:

• Outcomes (distinguished from outputs)

• Quality (as it pertains to all aspects of funded activity)

• Reach (into both communities of interest and communities of practice)

• Impact (in its direct, indirect and induced forms)

The next step will then be to deploy the definitions of the key concepts of the public value model consistently across all parts of arts and cultural policy.

2. Adopt for each public value dimension an ordered, one-dimensional scale (from which respondents choose one option that best aligns with their view on the characteristics of the funded activity - particularly the quality dimension)

3. Review, and where necessary recast, DCA’s overall policy frameworks. To develop the PVMF we have already begun this the process, as reported in Chapter Six. In light of the completed value framework DCA needs to ensure that its priority output areas are producing the full range of intended value, identifying in

the process what might be missing from the priorities and how they can best be articulated to ensure integration with the metrics of the PVMF

4. This review and recasting of DCA’s priorities will provide the platform for a parallel review of its Key Performance Indicators, with some necessary negotiation to ensure they include core public value metrics

5. Create a set of weightings to apply to the measures that reflect DCA’s policy intent for each funding program.

6. Align the funding application and acquittal forms to the new model

7. Consider options for a public value training program within DCA to ensure that the definitions are applied uniformly

8. Provide an effective communication strategy for public value awareness amongst key stakeholders and relevant interest groups

9. Explore how all funding, acquittal and assessment processes can be run effectively and efficiently online and through related media formats within two years

10. Create and maintain a database of pre-output (funding assessment) and post-output (funding acquittal) data that can be used for annual, aggregate public value assessments.

11. Create decision rules (which are specified in the policy framework) for the funding allocation process.

8 RECOMMENDATIONS AND IMPLEMENTATION ISSUES

42


12. Develop internal and external opportunities for longitudinal research and evaluation based on core data within the framework.

13. Harness the PVMF to a clear Gross Value Added (GVA) model.

The PVMF will allow DCA to tell a richer story of the economic benefit of its cultural investments. But in addition, DCA should take the opportunity the new system affords, to ensure that its reporting requests on funded organisations (in terms of net profits, FTEs, supply chain effects) are such that they can easily produce annual Gross Value Added calculations for the funded arts and cultural sector. In our peer review session with Professor Throsby, he acknowledge that the UK’s HMT Green Book Approach to GVA is becoming well understood in Australia, and now would be a good time for the arts sector to adopt a common GVA model. DCA could lead that debate inside WA and across Australia, driven by the implementation work on the PVMF.

14. Determine the public value Infrastructure in which DCA should invest

As we have noted in our description of the model, DCA’s investments will create a wide range of broader public value, whether in terms of educational attainment, social and community benefit, place making and wider economic benefits. The PVMF is not designed to capture all of these benefits through the reporting metrics, although it will provide part of the value story in these areas. Therefore we recommend that DCA develop a clear strategy for investing in the wider public value

infrastructure that will allow it to capture these benefits. This ‘infrastructure’ might include:

• A major public value study of key funded institutions. Our recommendation is that DCA commission such a study when it implements the new PVMF, in order to provide a baseline public value measurement at that point.

• DCA developing a contingent valuation methodology that it might use offer as part of a package of tools to allow funded organisations to generate a richer account of their institutional value, and proxies for public preferences and support.

• Commissioning joint evaluation work, in partnership with other key state departments (Education, Health), to better capture the value creating work of the arts in these areas. This could relate to both specific funding programmes, and the education work strands of the major stage companies and arts organisations.

8.2 IMPLEMENTATION CHALLENGES

The PVMF poses a range of operational challenges for the DCA. It is clearly stepping beyond our brief to work through all of the key operational and implementation issues, which will be addressed by DCA in the next stage of the PVMF project.

Clearly, at this stage we cannot provide a complete account of the challenges involved in implementing the PVMF. However, we can usefully illustrate the process and challenges that will be involved in developing the ‘nuts and bolts’ of the PVMF measurement tool, and we close with those observations.

43


The vision for the PVMF tool, currently in early stages of testing, is of an electronic interface integrating all of the existing measurements of financial and attendance data with a weighting framework, internal and public scoresheets and set of derived value outcomes for every DCA funding application.

As discussed within Chapter 3 of the report, instrumental value, intrinsic value and institutional value are three types of value that can be measured within the arts. Each of the values contains a wide range of metrics that must be collected in a user-friendly manner by the department, the arts companies and also the public. The PVMF tool needs the ability to bring together metrics related to each value, enabling all of the information to be contained in one place and displayed in such a way that it can be used for effective decision-making. The aim is to improve upon the limitations of the current system, in which methods of collecting and recording data are inconsistent across funding programs, often in hard copy and not input into a comprehensive database. This makes it difficult to compare the value of two applications when deciding on funding, or to compare the performance of a company in separate years.

8.2.1 Purpose

One challenge for DCA is to precisely determine the ultimate purpose of the PVMF tool. The primary value lies in collecting and collating data from a wide variety of sources according to a consistent method, using agreed and understood terminology. It can also be used to measure the value of an individual activity at application stage and following completion of the event, applying weightings to reflect the relative importance of different factors

(i.e. cultural tradition versus contemporary innovation). Consistent measurement should enable events to be compared with other events in the same funding stream (two visual arts exhibitions) or with events in different funding streams (such as performing arts). The PVMF tool could contain the ability to compare the total value of two funding streams, which would assist with setting initial fund allocations.

8.2.2 Variability

During development of the analytical model, testing of several funding applications identified a vast array of result combinations. This demonstrated that even within funding streams, two events could score very differently with respect to monetary revenue, attendance, educational value or perception of quality, and that generic decision rules cannot be applied across the board.

A challenge is to tailor the PVMF tool to ensure individual activities are weighted and scored correctly, while also simplifying the process. This will partly be a function of the electronic interface and underlying algorithms, which should enable the user to easily input specifications (such as choosing between a range of weightings), which will filter through the model to populate the data results and generate the required reports. It is also expected that there will be consistencies within the funding streams or activities, some of which are currently known intuitively as the basis for making value judgements. As more and more data is input, it will be necessary for the PVMF tool to identify and quantify these patterns, to enable decision rules to be simplified in the next round of funding.

44


The commitment to using the PVMF tool should not preclude DCA from using a wide range of expert practitioner input during the various stages of the assessment process, looking beyond the metrics and dashboard outcomes to check the veracity of the emerging outcomes for individual applications and add an important layer of expert interpretation to aid DCA’s final decision making process.

8.2.3 Instrumental Value

Instrumental value is defined as the direct utility gained by the public from their experience at the funded event, divided by the investment required to deliver the event. As a proxy, this is often equated to the price the public are willing to pay to attend the event. Sales value is relatively easy to measure and is a general basic indicator of the ‘success’ of an event. However, reach is also a key consideration in measures of both instrumental value and intrinsic value (measured alongside quality), which could create challenges associated with double counting.

A challenge associated with calculating instrumental value is that often arts organisations are funded on an annual basis (or triennially in the case of organisations such as WAB, WASO, WAO and Black Swan Theatre) rather than on an event basis, making it difficult to isolate and match funding to expenses and revenues from individual performances. This means that instrumental value is more likely to be measured for the organisation as a whole, while intrinsic value will involve scoring by the public on an event-by-event basis. Consistency across the measures of value will need to be resolved in order to calculate total value.

8.2.4 Intrinsic Value

Intrinsic value aims to measure aspects of an event that cannot be measured simply in dollar terms. This is particularly relevant in the arts, where cultural experiences can break new ground, exposing audiences to thought-provoking ideas or preserving traditional values. The first stage of this project has predominantly involved consultation, research and benchmarking to define and agree upon the set of parameters that comprise the intrinsic value of the arts.

In order to determine the intrinsic value of a particular funding event, the PVMF measurement tool must enable a score to be attributed to each of the parameters by the applicant and department representative (during the application stage and following the event) and the public (following the event). First the PVMF tool must contain an interface that allows the department to apply different weightings to the parameters to reflect their relative significance. Testing the model has highlighted the importance of this step, due to the wide variation in qualities associated with different applications and a desire to encourage diverse experiences.

8.2.5 Weighting

The measurement system itself is policy neutral. It is the weightings used to drive the system that need to reflect funder intention. Weightings already exist in notional allocations of funds. The intention of the PVMF tool is to make these explicit, and extending into the content of each program. This means that programmes will be characterised by what they are intended to achieve, such as global excellence, new artistic approaches or community connection.

45


Weighting must also be applied to the different scorers (applicant, peer, public). It is evident that there can be varying degrees of disparity between the views of the applicant and peer, which could reflect an applicant’s high or low self-belief (whether justified). It is likely that after several funding rounds, the model will identify consistencies in how applicants score themselves, which can be built into decision rules. For example, if applicants for the most part score themselves more than a certain number of points above the peer score, it can be recognised as a standard error that should not unduly affect the funding decision process.

8.2.6 Scoring

It is expected that the applicants will score themselves electronically on each of the quality and reach parameters (with the exception of audience number) to be assessed against the internal peer or panel scores. It could be possible to build in one or more decision rules at this stage of the process, where for example funding is automatically rejected if the panel score (and/or applicant score) is below a certain level, or the applicant is interviewed if the disparity between their score and peer score is too large.

To determine total intrinsic value, the public score - a vital component of public value management - must join the applicant and peer scores. Various measures for recording public scores have been suggested (including an iPhone App) to ensure the public can respond quickly and easily. Challenges may arise if there is a lack of response, which could be affected by the variation in experience associated with different cultural activities. Another issue may arise if the public does not appreciate what is “good” for them culturally or educationally,

however including the measurement of institutional value is intended to counteract this.

It is expected that the public assessment questions will align with the identified quality parameters, however they may need to be tailored to suit the audience. The reach parameters are unlikely to be assessed by the public, so attendance numbers or ticket sales (exposure to artistic activity) must be integrated into the PVMF tool at this point. Multiplying public quality scores by the number of audience members (reach) illustrates the total size of the intrinsic value derived by the public. If an average of all public scores is used instead, reach elements will need to be integrated elsewhere to ensure impact is properly accounted for.

Questions regarding reach that need to be resolved are related to whether the assessment counts percentage of the community of interest (CoI) that attend an event or the total number of people reached. Using percentage of CoI could favour events with smaller communities, even though in some case bigger CoI’s can indicate more interesting or relevant cultural activities. Events with limited seasons could also be measured as only reaching a small percentage of the total CoI, meaning that the popularity of the event is not reflected. Reach much therefore include a combination of both measures to ensure that both specialised and mainstream events are valued.

8.2.7 Total Value

The major challenge of the PVMF tool lies in the process of combining all three values to produce a total value – for the funded event, the funding stream or the arts in WA in

46


entirety. As discussed, instrumental value and institutional value both have the ability to be measured in monetary terms, while intrinsic value combines self, peer and public scores with reach measures to derive a total public ‘utility’. Although it is possible to assess ‘utility units’ per dollar of DCA funding - which turns an impact assessment into a value assessment - it is impossible to place a dollar value on the utility units.

Initial testing has explored methods for both combining the value measures and ranking applicants based on their performance in each category. For comparative purposes, percentage-based scores are likely to be necessary in order to account for differences in weighting (i.e. some parameters may not be measured which would impact upon absolute scores). It may be necessary or worthwhile to compare applications based on the quality and reach parameters that they are intended to achieve rather than within a broader category like performing arts. For example, this could involve measuring all of the ‘contemporary’ and ‘risk-taking’ events separate to the ‘traditional’ and ‘authentic’ events.

Absolute scores are also relevant for determining the total impact and value of a funding stream or program compared with another or over time, because they include a measure of the size of the impact. Including the size in calculation of total utility enables utility units per dollar of DCA funds to be calculated correctly, without favouring smaller (lower funded) organisations.

8.2.8 Next Stage

This initial testing and thinking around implementation issues will form a platform for the next stages of the Funding Process Review. Thorough testing and development of the PVMF tool is expected to raise and resolve many more questions and challenges, while also identifying additional benefits for a wide variety of users and uses.

47

A1

Contingent valuation is based on understanding what people would be willing to pay for a particular good or service, for example library provision or visiting a ballet performance. The techniques are based on constructing a hypothetical market for the non-market goods to be valued and then attaching prices to them by asking people directly about their willingness to pay or willingness to accept compensation for it. The method is summarised in Mourato et al14 discussion of how to apply contingent valuation to heritage:

‘The aim of a contingent valuation study is to elicit individuals’ preferences, in monetary terms, for changes in the quantity or quality of a non-market good or service, such as the recorded heritage. By means of an appropriately designed questionnaire, a contingent market is designed where the good or service in question can be traded.

The hypothetical market defines the good itself, in the case the recorded heritage, the institutional context in which it would be provided and the way it would be financed. A random sample of people is asked directly to express their maximum willingness to pay (or minimum willingness to accept) for a change in the level of provision of the good or service’.

This outlines the stages that make up the contingent valuations, stages where it is important to carefully develop the valuation question and the hypothetical market. The stages detailed by Pearce et al15 include making sure the purpose of the survey is made clear 14 Mourato, S., Pearce, D., Ozedemiroglu, E. and Howarth,

A. (2000) ‘Beyond ‘dusty archives’: The economics of preserving recorded heritage’ Cultural Trends 39 87-121

15 Pearce, D. ‘An intellectual history of environmental economics’ Annual review of energy and the environment 27 51-81

to respondents, understanding their general attitudes towards the good or service to be valued, understanding the respondents social and economic characteristics and exploring their current use (or non-use) of the good or service before valuation is discussed.

Expert opinion is important in the development of a contingent valuation (de la Torre and Mason16), as expertise with regard to the good or service to be valued is essential for the formulation of the valuation question and the context for the valuation (Mason16, Mourato14, Eftec17).

Once these stages are complete a valuation can occur. Placing a value on a good or service during a contingent valuation is a difficult task and requires well thought-out, well developed and very detailed questions to avoid the elicitation of ‘meaningless’ answers (Pearce et al15). Thus the valuation scenario, in which the hypothetical willingness to pay or willingness to accept will elicited, must involve a clearly defined good or service, a clearly defined way of delivering this good or service, it must detail the institution responsible for this delivery and the way payment will (hypothetically) be collected. Crucially Pearce et al15 stress the importance of the elicitation method, particularly as this aspect of contingent valuation has been subject to the most scrutiny within the economics literature.

However contingent valuation has not been widely adopted for valuing culture in the UK and where it has been used, work has focused on the historic environment. However 16 de la Torre, M. and Mason, R. (2002) ‘Introduction’ in de la

Torre (2002) Assessing the value of cultural heritage Los Angeles: The Getty Conservation Institute pp 1-5

17 Eftec (2005) Economic Valuation of Environmental Impacts in the Severly Disadvantaged Areas Eftec: London

APPENDIX 1: CONTINGENT VALUATION METHOD

48


there have been two major studies with the museums, Libraries and Archives (MLA) sector, a valuation of the British Library and a valuation of Bolton’s museums, libraries and archives service.

Most useful for this context is Jura’s18 valuation of Bolton’s museums, libraries and archives. The work built on Spectrum’s work for the British library, which used contingent valuation to estimate that the British Library’s services provided £363 million of value to both users and non-users, as compared to its £83 million of grant-in-aid. Using contingent valuation allowed Jura to value both the costs and benefits of the MLA service to the local community in Bolton, showing how both users and non-users of the service valued this provision at £10.4 million, as compared with public funding of £6.5 million.

Jura’s work is of particular interest as it illustrates the benefits and problems of adopting stated preference valuation techniques. On the one hand the figures produced were robust and useful for cost-benefit analysis of the returns generated by public funding of the museums, libraries and archives in Bolton. Indeed a key conclusion of Jura’s study was that contingent valuation was appropriate to value a local museums, libraries and archives service. However the methodology was subject to the range of interrelated problems, philosophical as well as practical and technical.

18 Jura Consultants (2005) Bolton’s Museum, Library and Archive services: An economic valuation Edinburgh: Jura Consultants

49

A2

APPENDIX 2: METRICS BY DCA FUNDING PROGRAM

50


51

A3

APPENDIX 3: DCA STRUCTURE AND DATA

52


53

A4

Subject: Testing Workshop DSD Staff

Date: 6 December 2011 1pm

Attendees: Marty Cunningham, Rebecca Sheardown, Ricky Arnold, Liz Spencer, Karin Burrill, Jane Mitchell, Lyndsay Feltham, Michael Chappell, Jenny Nichol

Subject: PVMF Testing Workshop 2 DCA Peer Panellists

Date: 9 December 2011 9am - 12pm

Attendees: Marty Cunningham, Rebecca Sheardown, Helen Turner, Jonathan Mustard, Lee Kennedy, Bonnies Davies, Michael Chappell, Jenny Nichol Apologies: Michael McCall, Ionat Zurr

Subject: PVMF Peer Review DSD Strategic Managers

Date: 27 Febreuary 2012 2:30pm

Attendees: Jacqui Allen, Colin Walker, Marty Cunningham, Karin Burrill, Liz Spencer, Rebecca Sheardown, Michael Chappell, John Knwll Apologies: James Wells, Lyndsay Feltham, Chris Penwald, Ricky Arnold

Subject: PVMF Peer Review DCA Executive Representatives

Date: 28 February 2012 10am

Attendees: Allanah Lucas, Jacqui Allen, Chris Penwald, Rebecca Coakley, Margaret Butcher, Colin Walker, Eric Isailovic, Rebecca Sheardown, Michael Chappell, John Knell Apologies: Ricky Arnold

Subject: PVMF Peer Review - ‘All at DSD’ staff and DCA Managers

Date: 28 February 2012 2:30pm

Attendees: Jacqui Allen, Colin Walker, Marty Cunningham, Rebecca Sheardown, Alana Culverhouse, Ali Martin, Alysha Worth, Annie Thompson, Caroline O’Neill, Cathy Driver, Chris Penwald, Daevid Anderson, David D-Arcy Burke, Diane Johnston, Elaine Seymour, Ellen Richards, Fiona Nelson, Jaren Herman, Karin Burrill, Kate Bird, Kathleen Toomath, Kristine Genovese, Marit Kloostra, Roz Lipscombe, Tony Loiacono, Pam Svenningsen, Carl Pekin, Hannah Eames, Arthur Nastos, Michael Chappell, John Knell Apologies: Kate Jones, Lyndsay Feltham, Jane Mitchell, Linda Tavelli, Michelle Broun, Ricky Arnold, Eric Isailovic, Aly May, Carl Gopalkrishnan, Liz Spencer, Pete Guazzelli

Subject: PVMF Peer Review - Focus Group 1

Date: 29 February 2012 11am - 1:20pm

Attendees: Jacqui Allen, Marty Cunningham, Rebecca Sheardown, Simon Clarke, Pilar Kasat, Jim Cathcart, David Doyle, Monique Douglas, Dan Minchin, Michael Chappell, John Knell Apologies: Alice Lee Holland

Subject: PVMF Peer Review - Focus Group 2

Date: 29 February 2012 2pm - 4pm

Attendees: Marty Cunningham, Alysha Worth, Lyndsay Feltham, Aly May, Phillip Mitchell, Jude Van der Merwe, Marco Marcon, Jeremy Bean, Clem Rodney, Suzie Hazlehurst, Alan Payne, Andy Farrant, Michael Chappell, John Knell Apologies: Terri-Anne White, Marcus Canning, Ben Burgess

APPENDIX 4: MEETINGS AND CONSULTATION

54


Subject: PVMF Peer Review - Working Group

Date: 1 March 2012 10am

Attendees: Jacqui Allen, Marty Cunningham, Oliver Gaty, Ricky Arnold, Rebecca Sheardown, Michael Chappell, John Knell Apologies: Colin Walker, Chris Penwald

55

A5

APPENDIX 5: MINUTES

56


PVMF Testing Workshop DSD Staff Minutes 06/12/11 page 1 of 3

Development and Strategy Directorate

Minutes Gordon Stephenson House, 140 William Street, Perth WA 6000

Tel: 6552 7300 Fax: 6552 7301 Email: [email protected] www.dca.wa.gov.au

FILE NO: 11/135

SUBJECT: Testing Workshop DSD Staff

DATE: 6 December 2011 1pm

ATTENDEES: Marty Cunningham, Rebecca Sheardown, Ricky Arnold, Liz Spencer, Karin Burrill, Jane Mitchell, Lyndsay Feltham

Michael Chappell, Jenny Nichols

APOLOGIES: -

Tabled documents; 1. Pracsys Form A and B 2. DCA PVMF measures and corresponding data collection Q = Questions from the DSD staff A = Answers from Pracsys

Item Summary of discussions ( * indicates action) Actions

1. Introduction Michael Chappell to provide background and introduce Form A and B. MCh provided an overall introduction to this stage of the project and practicalities of working the measures in to the DCA funding processes. It is estimated that there will be a post output to the testing stage. In regards to the Public measurement there is consideration to create an iPhone Application to collect data. Q: JM questioned if these measures provided would be the ones on the iPhone App? A: MCh said they would be and that the language would have to be user friendly and brief. MCh suggested the three tiers of assessment be kept in perspective while reviewing the definitions, being; *

1. Self (eg: Applicants) 2. Peer (Peer Panellists, PO’s) 3. Public (audience, general public)

RS - Include the three tiers of assessment on the DSD definition comments.

2. Public Value MCh showed the Public Value Framework Design diagram and explained where in the process that the


Framework Design measures of Quality and Reach. RA identified that Value required a lot of work eg: writers R&D to produce and provide draft does not always provide an end product. Potential also needs to be clear. JN suggested the post output can identify Value better. The section of Impact and Value is being undertaken by John Knell in the Peer Review. John Knell has provided a draft report last week. Benchmarking: The PVMF is the beginning of collection and benchmarks will slowly build. The data will be a rich source from the three tiers of collection. It is possible there may be some benchmarks provided in the International Peer Review from John Knell. Benchmarking will also be determined when the framework is implemented.

3. Definitions Q: The group considered at length that the language of the Definitions could become a barrier for Applicants. A: MCh added that the Weightings of the measures would assist. In relation to Creating Value there are two overarching values of Creativity and Engagement. Q: There was discussion that ORIGINIALITY and RISK are the same? A: When the definitions were first prepared there were many more and Originality replaced Uniqueness, Newness and Novelty. INNOVATION – change the word “innovative” to read “creative”.* Q: JM asked if these Definitions replace DCA’s criteria? A: MC explained that the Funding Process Review identified the need to align funding criteria with Creating Value. Part three of the Funding Process Review is the alignment of DCA’s funding programs to the Policy. MCh added that the alignment options are;

a) Change criteria to match the Policy b) Change the Policy to match the criteria c) Or you acknowledge there is no link between

both In this case it will be the weightings that have the most impact on the definitions. REACH – we already have ways of measuring this but we do need some subtle differences. Q: This raised the longevity and survival of the piece of work eg; a novel that becomes a script then to a movie.

Add PLATFORM in Reach – longevity of influence and importance to Community of Interest and Community of Practice.*

Pracsys will recommend in their final Report that there is greater budget allocation for data research and that there is also a traditional Qualitative Data Analysis

a) INNOVATION – change the word “innovative” to read “creative”.*

b) Add PLATFORM in Reach – longevity of influence and importance to Community of Interest and Community of Practice.*

c) Add LEVERAGE (under Practice) – ability to attract investment from non-DCA sources.*


(QDA) Add LEVERAGE (under Practice) – ability to attract investment from non-DCA sources.*

4. Process Discussion on variances on Scores. MCh outlined the distinction with which variances have been identified; a) Applicant thinks they are good – Panel says they

are bad b) Applicant thinks they are bad – Panel thinks they

are good c) Applicant and Panel both think they are good d) Applicant and Panel both think they are bad Based on the weightings applied on these scores there would be a final score. The value of the funding determines the level of acquittal. Q: RA understood that we were reviewing the definitions for Quality and Reach, but was seeking clarification on what information was available yet regarding the full public value equation including Impact and Value. RA considered that understanding the full equation would influence the measures being developed in Quality and Reach. A: MCh explained that the workshop was only regarding the definitions for Quality and Reach and that the full public value equation is being developed in the final report and by peer review by John knell. Q: Do the Definitions Evolve? A: Fundamentally the concepts of the definitions remain. The weightings can change, but if the definitions change drastically then there can be trouble with comparing data.

5. Peer Panel workshop

Methodology will include: • Definitions – and the importance that the metrics are

not criteria • Hypothetical application testing

6. Conclusion Pracsys will be modifying the Definitions based on the two workshops and will come back in the new year to propose the integration into the funding processes.

PVMF Testing Workshop 2 DCA Peer Panellists 09/12/11 page 1 of 3




FILE NO: 11/135

SUBJECT: PVMF Testing Workshop 2 DCA Peer Panellists

DATE: 9 December 2011 9am-12pm

ATTENDEES: Marty Cunningham, Rebecca Sheardown, Helen Turner, Jonathan Mustard, Lee Kennedy, Bonnie Davies

Michael Chappell, Jenny Nichols

APOLOGIES: Michael McCall, Ionat Zurr

Tabled documents; 1. Pracsys Form A and B 2. Example Application and Acquittal – Rebecca Baumann Q = Questions from the DSD staff A = Answers from Pracsys, unless identified by other


1. Introduction MC provided an overview of the relationship of the PVMF and Creating Value and clarified how the peer process is fundamental to how funding is decided.

-

2. Background MCh presented the Public Value Measurement Design diagram and gave overview of the measurement framework. MCh explained that the DCA require a “smart system” – closed ended loop that feeds data back into itself, which it currently does not have. Q: Regarding the timeline and implementation. A: Testing stage now, with the possibility of it being integrated into some panels in 2012 as a further test before if it is determined to proceed.

-

3. Definitions QUALITY Q: The term “New” was discussed at length. The distinction works do not have to be new all the time. That there is value in building upon an idea and building it up A: Consideration to replace the repetition and the word new in the definitions of Imagination, Originality

Consideration to replace the repetition and the word new in the definitions of Imagination, Originality and


and Risk. * Q: JM- The measures seem too much like a checklist, rather than an environment or economy. A: Agrees, but clarified that the form is not a checklist. Q: JM propose “Inventiveness” as an extra term? A: The group feels it may be covered by “Innovation” but accepts that it reduces the “newness” of practice. Q: BD asks where is “Artistic”? A: MC confirms that DCA use “Creativity” to cover the breadth of all practices and “Creativity” was seen as more encompassing to all artforms. It includes commercial outcomes as well. Q: JN asked LK directly what he thought about “Newness”. A: LK likes new work and innovation. However thinks what is presented is too complex and that its takes away the individual discussion of each application. ISSUE: The group unanimously agree that there is a major concern that in its present format the measures look too much like criteria and it would be a major barrier to potential Applicants.* Group suggested that the measures be tested through each artform* LK still sees the community applicants would find this type application process difficult. BD suggests under “Innovation”, that the word ‘transform’ be changed to ‘realise’. Group agreed.* LK suggests the whole system should be further simplified to make it more flexible. JM stills disagrees with “Innovation” Suggested changes would be;

1) “Transformation” to something like “Impact” 2) “Innovation” to be altered

The group were advised that “Authenticity” had come from the Staff workshop prior in the week. MC suggested that there were a number of points made that were focusing on the parameters of ‘authenticity’, ‘originality’ and ‘innovation’ that seemed to point to aspects of ‘practice’ – how an artist goes about their creative process. REACH JM suggested that “Leverage” doesn’t need a scale. JM showed an example from a recent university study www.dancingbetweendiversity.com

Risk.*

ISSUE: The group unanimously agree that there is a major concern that in its present format the measures look too much like criteria and it would be a major barrier to potential Applicants.* Group suggested that the measures be tested through each artform*

BD suggests under “Innovation”, that the word ‘transform’ be changed to “realise”. Group agreed.*

4. Process MCH explained at what points the data would be collected, • Starting from the Application, then Peer


Assessment and finally at the Acquittal stage. • Panellists should forget the rating scale • Pracsys will be recommending a framework and it

will be up to DCA to collect the data. • The suggestion of a simpler rating system would

be reviewed, such as a A,B,C or Green/Yellow/Red and to include a NA category.

• It is understood that the scale and its presentation in the process needs much more work.

There was discussion regarding the Peer Assessment and MCh mentioned the online acquittal process and also the possibility of an iPhone App. • BD identifies the iApp could bring unsolicited ‘hate

mail’. LK raised the applicability to projects that are providing mentorship. LK has also previously asked DCA for Acquittal from previous applicants who may be submitting a new application to see the second stage of the product. However, PO’s have suggested these may cloud the process and have not allowed it. It was acknowledged that the ongoing work of the peers for reviewing work was critical. The Peer Review methodology needs to simple and clear.

5. Workshop review

MCh identifies the practicalities and implementation of the framework needs to be less onerous and cannot cause stress to potential applicants.

6. Conclusion

MC gave an overview of the Funding Process Review. The research at present is about undertaking macro level data of arts and culture and the broader impact of arts and culture on the community.

DSD Strategic Managers 27.02.12 page 1 of 3




FILE NO: 11/135

SUBJECT: PVMF peer review DSD Strategic Managers

DATE: Monday 27 February 2012, 2.30pm

ATTENDEES: Jacqui Allen, Colin Walker, Marty Cunningham (arrived 3pm), Karin Burrill, Liz Spencer, Rebecca Sheardown (minutes)

Michael Chappell, John Knell

APOLOGIES: James Wells, Lyndsay Feltham, Chris Penwald, Ricky Arnold


1. Introduction JA – Asked for a general overview on the status of the PVMF. MCh – Making sense of public value and seeing how Quality and Reach have different priorities within the Creating Value framework. Developing the framework is the simple part as it will be the implementation that will be the most difficult.

2. Implementation of the PVMF

JK – Noted that commenting on the implementation within the final report was important so the sector and stakeholders may see how large a project it is and key decisions to be made DCA should not underestimate how large this work is on an international scale to be developing a measurement system. JA – At this point designed a flow chart that represents the internal and external flow of accountability and transparency through the Portfolio that flows into the Community. Public Value Delivery


3. Development of PVMF Final Report

CW – International agencies such as ACE and Australia Council are arms-length funders and there needs to be some commentary in the final report why DCA is different to those. It needs to be clear that the Minister signs off on the final decision.*

JK – As more discussions are held it becomes clear how the model keeps evolving. The base for the model is Accountability which affects the Portfolio decisions. Confirms they will make the decision making aspects clear in the final report. DCA need to realise that when it develop Creating Value it was asking for a rigorous PVMF system. JA – The PVMF needs to level the disparity between organisations. Needs to move discussion beyond personal patronage. MCh – Currently there is a lack of definition for infrastructure in Creating Value and there is a common language needed. JK – Suggest making progress in the final report by addressing; • What does the model mean to Australia?* • What opportunities does this piece of work do for the

sector?* If the model works it will be something great for DCA and WA. JA – Sees how the prospect to promote this “great opportunity” and that the measurements DCA provide should influence the Organisations KPI’s. MCh – Broader measurements to define an objective system to get away from the subjective decisions. CW – Requests that the final report states more clearly where the data points are for the Peer Assessment process and to show the efficiency of the model.

Pracsys to comment on the DCA’s difference to arms-length funding in Final Report.

Pracsys to address what the PVMF means to Australia and what opportunities there are for the sector.

4. Accountability and Transparency

With this new model DCA will have the evidence to prove the rigour of measurement and economic impact of the arts to Government.

5. Key Messages Risks; • That Organisations are not happy with the PVMF and

refuse to engage. • Organisations are not expected to meet all of the

outcome areas and metrics. Communication messages; 1. Need a clear upfront message 2. Narrative that sells to the different stakeholders. 3. Need for stronger Collective Message MCh – identifies there are three position papers needed for the following stakeholders;

1. Minister’s office 2. Public 3. Internal > process changes

DCA will develop these position papers.

DCA to develop Communications Plan; inc Risks – • That

Organisations are not happy with the PVMF and refuse to engage.

• Organisations are not expected to do fulfil all of the measures.


JA – It is critical that there is positive response from all DCA business managers of the new framework. “Multi-point Subjectivity” – a new term to describe how the PVMF provides a system that attempts to measure a process that has historically been subjective. CW – Theoretically, once the PVMF is implemented there will not be any other system for comparison. The model should be able to test policy intentions and assumptions and be explicit. Messages developed collectively; • Implicit and Explicit range of measures. • Developing Trust

Communication messages; • Need a clear

upfront message

• Narrative that sells to the different stakeholders.

DCA develop three position papers for 1. Minister’s

office 2. Public 3. Internal >

process changes

6. Australia Council ‘Culture Count: Measuring Cultural Value’ seminar.

Australia Council ‘Culture Count: Measuring Cultural Value’ seminar. • It would be beneficial for DCA to work collaboratively

on the federal counterparts and the Organisations we co-fund.

• JK recommended that DCA visits the Australia Council Review survey online.

7. Meeting context for scheduled week

The group discussed the outline of content for further meetings during the week with the representatives from; • the Executive, • DSD Staff and Managers; and • Focus Groups with Organisations. Stronger up font pre-amble and context. Selling narrative; • What this means for all players, • The rationale for change, • Refining efficiency of data collected. Key questions for academic reviewers; • Do you buy into the model • Do you think the process is rigorous enough. Key questions for DCA • How will the model impact on internal resource

allocations?

PVMF peer review DCA Executive representatives 28/02/12 page 1 of 2




FILE NO: 11/135

SUBJECT: PVMF peer review DCA Executive representatives

DATE: Tuesday 28 February 2012, 10am

ATTENDEES: Allanah Lucas, Jacqui Allen, Chris Penwald, Rebecca Coakley, Margaret Butcher, Colin Walker, Eric Isailovic, Rebecca Sheardown (minutes)


APOLOGIES: Ricky Arnold


1. Introduction JA introduced the consultants to the DCA representatives and explained the PVMF will require the commitment of all business operations throughout DCA.

AL confirms this system is important for Government to see what their departments deliver.

JK provided background of his experience and knowledge and confirmed DCA is the first agency globally to introduce a working public value system and this has generated a lot of interest from the international arts sector.

MCh provided background on his experience as an economist and strong interest in the arts.

2. Context JA - Arts funding is presumed that there is no return on investment. Need to convince Government that we are smart and can measure our ecocomic impact.

The assessment framework is objective and subjective.

3. Diagram: Creating an operable measurement framework

MCh spoke of that the precision is in the language and that the meadure need to remain consistent as well as the language.

MCh presented diagram: Creating an operable measurement framework and discussed the disconnection between the Applications and Aquittals process and how the new model needs a closed loop of reporting.

4. Diagram: Creating a learning system

MCh presented diagram: Creating a Learning System Showing how the system learns, making self-assessment a more meaningful tool.

PVMF peer review DCA Executive representatives 28/02/12 page 2 of 2

5. Diagram: Public Value Measurement Framework Process

MCh presented the logic framework: Public Value Measurement Framework Process Diagram. MCh explained that the Weightings change within the model but not the Definitions. JK explained that it is not expected that one Organisation would fulfil or conform to all of the public value measures. Q: EI – how do we convince organisations that there is not one score rating? A: JK – DCA will be clearer than it’s ever been before as to what it expects and require from the organisations. MCh - Common language is very important and most terms come from venture capital. The development of the Definitions was a long and rigorous discovery from working sessions with DCA staff and then tested again through staff and peer panellists. AL – considers the PVMF will provide the sector with more clarity and certainty because it allows precision responses.

6. DCA Key Performance Indicators

Q: CP – How does this work with the alignment and review of DCA’s Key Performance Indicators (KPI’s)? A: MCh – impact is divide by Outputs which delivers a value amount.

7. Strategic Planning

MCh – if DCA shows DTF that there is a new system and change management structure then there would be support from DTF. Q: EI – how will this change DCA? A: JA – Allows us to give a very clear decision making tool and requires the support from the IT systems. Q: CP – Impacts on general operations and what capacity does DCA have to accommodate this? A: JA – Need the commitment from all of DCA regarding the implementation before it is done. More resources and a restructure may be required. A strategic planning session is necessary for the impact of the PVMF and other Category A projects currently in development.* MCh suggests the iPhone App would be a very simple trial platform to start testing the PVMF. EI – suggests that one or two organisations be tested through the new framework before it is implemented.* agreed Q: TL – has this model been considered in the development of the Online Grants Management System? A: MCh met with the consultant who designed the statement of requirements for the new system.

DCA need to hold a strategic planning session for the impact of the PVMF and other Category A projects currently in development.*

DCA will test one or two organisations through the PVMF before it is launched to the sector.

8. Conclusion MCh explained the visits in the eastern states is to get more academics and peers to review the concepts of the PVMF for feedback and endorsement.

PVMF peer review - ‘All at DSD’ staff and DCA Managers 28.02.12 page 1 of 2




FILE NO: 11/135

SUBJECT: PVMF peer review - ‘All at DSD’ staff and DCA Managers

DATE: 28 February 2012, 2.30pm

ATTENDEES: Jacqui Allen, Colin Walker, Marty Cunningham, Rebecca Sheardown (minutes), Alana Culverhouse, Ali Martin, Alysha Worth, Annie Thompson, Caroline O’Neill, Cathy Driver, Chris Penwald, Daevid Anderson, David D’Arcy Burke, Dianne Johnston, Elaine Seymour, Ellen Richards, Fiona Nelson, Karen Herman, Karin Burrill, Kate Bird, Kathleen Toomath, Kristine Genovese, Marit Kloostra, Roz Lipscombe, Tony Loiacono, Pam Svenningsen, Carl Pekin, Hannah Eames, Arthur Nastos


APOLOGIES: Kate Jones, Lyndsay Feltham, Jane Mitchell, Linda Tavelli, Michelle Broun, Ricky Arnold, Eric Isailovic, Aly May, Carl Gopalkrishnan, Liz Spencer, Pete Guazzelli


1. Introduction JA introduction to staff about the status of the PVMF.

MCh and JK provided their backgrounds of experience and gave an overview of the previous testing meetings held in October 2011.


MCh/JK provide an overview how the public value Definitions provide a measurement system for the Applications and Acquittals process and how these then assist with funding decisions.


MCh/JK gave an outline of how the new system will learn from itself through the three levels of self, peer and public assessment.


MCh/JK presented the logic framework model and how the measurement system will work.

5. Form A & B MCh/JK distributed Form A & B that show the current Definitions. They explained how the Definitions had been developed and then presented internally and with selected peer panellists. The definitions for the

Pracsys to integrate how Partnerships with no dollar value

PVMF peer review - ‘All at DSD’ staff and DCA Managers 28.02.12 page 2 of 2

framework have come directly from the Creating Value policy framework. Q: Could the consultants explain the term ‘Community of Interest’. A: MCh gave an example where if he attended a performance then he is part of an audience, but at the same time he is representative of a demographic of another 50,000 people within the community. Q: Designer Fashion/Marit Kloostra – how do these measure work with the long list of measures already identified under the creative industries (CI) program? A: MCh responded that not all the measures will fit each program equally and they are not expected to. However the measures need to be a standardised system. JK says the CI were very much considered in development of the definitions and by continuing to work with the expertise of staff it will make the system much stronger. MCh identifies that if there is state funding provided to CI then there is a policy responsibility. JK suggests that ‘Platform’ is a good term that applies to longevity and measuring CI. Q: How does Skills Development fit with the new scoring system? A: JA stated that the policy Weightings will determine how there different measures are fulfilled. Q: Is anyone outside of the arts sector surveyed? A: Anyone can be surveyed through the public survey assessment. Plus, Organisations expect DCA to collect data from them as they are really co-creators. Public surveying is important for Capacity Building. Q: How would Partnerships with no dollar value be measured? A: That is a very good query and Pracsys will investigate this further.* ‘Leverage’ could be expanded to encapsulate this. This model is an attempt to show how the rigour with which it can be presented to Government/DTF.

can be measured. Possibliy to expand ‘Leverage’ to include value of networks and partnerships.

6. Conclusion • There are more discussions that the consultants are having before the Definitions are finalised.

• We are not trying to move away from peer assessments; we are working alongside an objective assessment, which has been branded as ‘Multipoint Subjectivity’.

• The new system will be to be positively endorsed by other directorates across the portfolio.

• Later this week will be Focus Group with organisations regarding the new system and the Definitions.

PVMF peer review - Focus Group 1 minutes 29.02.12 page 1 of 3




FILE NO: 11/135

SUBJECT: PVMF peer review - Focus Group 1

DATE: 29 February 2012, 11am-1:20pm

ATTENDEES: Jacqui Allen, Marty Cunningham Rebecca Sheardown (minutes)

Simon Clarke, Pilar Kasat, Jim Cathcart, David Doyle, Monique Douglas, Dan Minchin


APOLOGIES: Alice Lee Holland

Tabled documents; • Form A & B Q = Questions from the sector A = Answers from Pracsys and John Knell


1. Introduction JA provided and introduction to the representatives from the organisations. • This work has come from the launch of Creating

Value two years ago and since then there has been extensive research to get to this point.

• DCA is keen to work with the Organisations to develop this work.

• Today session will be about testing the draft metrics in the PVMF.

MCh and JK provided each of their backgrounds for those who had not met them before. Roundtable introductions were done.

2. Opening comments

MCh & JK posed some questions to the group as a way of setting the context for today’s session. • Will this framework allow the Organisations to do

their work within this model? • This work is powerful and it is great that DCA are

building this new system. • Public value frameworks and measurements have

been discussed for a while internationally. • Any measurement system that is developed needs to


be valuable to the organisations and not just responding to the Funders requirements.


MCh/JK provided an overview how the measurement of public value should assist with funding decisions. • Quality and Impact have been widely used terms in

public value theory. • MCh states that Creating Value is a highly respected

public value policy in Australia and stands up quite well

JA stated how the DCA system is subjective and it needs to be more objective and DCA is heading towards Multi-point Subjectivity.


MCh provided an example to illustrate the learning system. • Example - One of the funding reports that includes

data; all the paper goes into a hardcopy file and is not seen again for another 12 months. This is common globally. Plus the Application process has not changed for approximately 15 years and only collects Self and peer data.

• In the future there will be Self, Peer and Public measurements and this will help to streamline the Application and Acquittal processes.

Q: DD – Are there cross government impacts? A: MCh – yes there are. Comment: DM identified that if organisations can demonstrate Value to government and DCA then you can convince other industries for leverage.


MCh explained the public value framework model • Described how the definitions under Quality and

Reach had been rigorously researched. • Ways of collecting public review could be done by

creating an iPhone App (as an example). JK identified how Impact can be used to target Education, Heath, Tourism, Place Making and Creative Economy. MCh explained that if you divide Impact by Inputs then you have a basic economist value and this then gives decisions back to the portfolio. Group discussion: agreement that all could see how this work to measure public value should garner more government support.


JA explained to the group that the Weightings are not equally rated across all programs and that his system is not a popularity vote.

MCh acknowledges that the DCA system needs to be very simple and clear to alleviate demands on the organisations given the variety of funding sources they engage with.


7. Form A & B Initial reaction by the group was positive. Q: DM – what about Economic Output? A: MCh – Creative industries have commercial outcomes but are funded for market failure. Q: PK – Sees there are risks with not ‘telling the stories’ politically. A: JA – stated that there will be consistency in the reporting and in the implementation and supported by research. Moves decisions from patronage to public value. A: JK – This work supports the expert knowledge provided by panellists that will still be provided.

8. Conclusion Members of the focus group are asked to send any more feedback to Marty Cunningham via email.* JA says this discussion will continue and there are many other related projects ongoing in DCA. There will a trial period for the system to be thoroughly tested ahead of any rollout to the sector.

Members of the focus group asked to send feedback to Marty Cunningham via email.

PVMF peer review - Focus Group 2 Minutes 29/02/12 page 1 of 5




FILE NO: 11/135

SUBJECT: PVMF peer review - Focus Group 2

DATE: 29 February 2012, 2-4pm

ATTENDEES: Marty Cunningham, Alysha Worth (minutes), Lyndsay Feltham, Aly May (observer)

Phillip Mitchell, Jude Van der Merwe, Marco Marcon, Jeremy Bean, Clem Rodney, Suzie Hazlehurst, Alan Payne, Andy Farrant,


APOLOGIES: Terri-Anne White, Marcus Canning, Ben Burgess

Tabled documents; • Form A & B Q = Questions from the sector A = Answers from Pracsys and John Knell


1. Introduction MC gave a brief overview and context: • Policy Framework context – value of the delivery of

arts and culture – definition of the broader value • Common language • Credibility – must be common language and

accepted by the sector • Critical to the success of the framework MCh and JK provided their background. Roundtable introductions. MCh and JK provided an overview of the uniqueness of the project • It is not about ticking boxes • Long term measurement system = needs to reflect

the whole sector therefore important it is a co-produced framework, explaining the value of the meeting.

• No need to distort art ambition to make the case for funding

• Common shared intention • Shared understanding of the terms and the act of

creating a shared understanding during this


process


Fundamentals regarding communication of public value is language. Explanation of diagram; Creating an operable measurement framework Quality, Reach, Impact and Value • Inconsistency in the measure in application and

acquittal process • No metrics that are quantified No Questions from attendees


Comment MM: currently there is much more scrutiny in applying for the grant than the acquittal process: good proposal get the grant but no assessment after the project (verbally trying to understand the framework)

4. Diagram: Public Value Measurement Framework Process Diagram

Difference of outputs and outcomes Outputs – bums on seats Outcomes – results/value State context: Fund Studies, infrastructure, management etc. People have no access to this information and is currently not useful or being used JK - There is a broader value of the whole sector and collaboration of other agencies to measure value. E.g. education and DCA collaborating. Q: MM - how does DCA justify putting money into combined agency research when the other agencies receive higher budgets and have more money to throw around? A: MCh - it is about the broader benefit and DCA will have further information into arguing value and more money. They can also argue for combining funding for a collective outcome. Cross agency studies are not done in Australia. Only available in the UK and Europe. This framework will include these kinds of studies MCh - User review is an everyday thing. Why aren’t we doing reviews through apps in the arts? (group agreement) Evaluations to be compared to pre and post output and provide a variety of opinions. If you gather enough data points and sample size, then the information is of value. We want to embrace the peer assessment and industry knowledge. JK: Measurement system. The responsibilities become clearer. This framework will sharpen DCA’s point of view and will therefore be able to accommodate others effectively. MCh – Weightings are important because we don’t want all types of the arts to be the same. We need to choose the parameters depending on the art form etc.

Focus Group participants are asked to send any feedback to Marty Cunningham via email.


JK – No distortion from the arts orgs applying for funding They do not have to apply and be assessed against all parameters. It is very important when looking at the diagram and definitions that the measurement system needs to be policy neutral. This means it can/could be used by anyone, including the Australia Council and other government state agencies – there is no policy agenda when developing this framework. Q: AP is there a possibility that Quality could be changed because of Reach? A: MCh – It does not remove expert judgement. Both art forms and orgs have its strengths and weaknesses. The framework will provide a better evidence base to make decisions. If the framework works, the sector should not see that the DCA is not performing according to what was intended.

5. Form A & B Q: PM - Creative Economy: what does this mean? A: JK - Treasury want a number – economic effect. Creative economy is different: public funding of the arts is different. You want the metric framework around leverage and platform to tell the value story. Comment: CR – Authenticity – what this means to the work they do (at YY) is important. He agreed that this is different to ‘distinctivness’. MCh - There will also be specific research and studies DCA will commission along side these in order to get various data. Q: MM – Has concerns about education and health and combining with these agencies. How strong would the argument be if DCA was to combine with the health department? Who is going to receive this message from the information obtained from this research? A: JK - Given there is international impact of arts in society. The value system needs to display these values including these assessments. DCA could go to the agencies with our new language and request a combined budget. MCh distributed Form A & B • We need to make sure the definitions have

universality. • There is a possibility to change Authenticity to

Distinctive but the meaning would be skewed and change.

• Authenticity is of culture of place, of WA’ness. • Clear understanding of authenticity and how the

definitions and framework will work in terms of telling aboriginal stories

Q: Question regarding Community of Interest What is this? A: MCh gave an example where if he attended a


performance then he is part of an audience, but at the same time he is representative of a demographic of another 50,000 people within the community. It is about the size of the Community of Interest rather than the bums on seats. JK - Discussion at the DCA staff session yesterday: the point was raised about skills development How are they adding developing skills? Comment: MM - public assessment: this is much better compass than the Urbanspoon example. MCh - The Definitions sheet and diagrams is a backend database and not what the public or peer would see. They will all have their own weightings depending on the criteria for funding. We will only collect the information of what we will use. Additional resources will be needed in DCA to develop and implement this framework and provide the review and data collection. JK – Invites the group to take some time to absorb and come back to DCA with feedback. * Consider what the prompt questions would be. Quality - High and low arts. Traditional assessments in artistic performance. What is important if we measure the value the community projects will score higher than the high arts. No quality connection without the quality of art Comment: MM - Important quality in the assessment in practice. For people in WA out of Perth the numbers will be less. A: MCh - This is why the size of Community of Interest is important. JK – Wellbeing; we know you will get bigger impact if you are offering participatory than watching Quality of experience of audience matters. Comment: AF - Local communities and local councils are struggling to understand the value of funding the arts in these areas. They are not receiving the money in council discussions. Q: SH: The increase in resources needed was mentioned. The people with less capacity will need more support to understand this. Will the department be able to provide this? A: MCh - DCA have a determination to make this happen. DCA will be meeting with Australia Council and academics too get industry support. Comment: AF - The communications strategy around this is going to be enormous. Because of the relationships DCA has with the agencies Australia Council could be your partners to share this. Third party endorsers. Q: JVDM – Is positive about the framework. But would like to know when will it be finished, when the end date will be and what is the implementation


timeframe? A: Pracsys is at the end of developing the framework project plan. DCA will be planning the timeframe and deciding on the final recommendations and implement in due course. There are some immediate opportunities to start testing data the ‘Soft way’ by using iPhone Apps. It is up to DCA to make decisions of how will the data generated will be handled. Q: JVDM - It has been two years since the Creating Value launch. Will the online PVMF system be launched within two years? I am not trying to be critical of the time taken but I think it would help the whole sector to give a timeframe and then work to it. Q: How does this relate to the connection of treasury? A: Treasury think DCA is a cost only agency. The framework gives a benefit value. Other agencies have a clear way to calculate benefit in terms of economics. Q: Are you trying to tap into treasury language? Or are you saying this is our measurement language and take it? A: This is about the long-term discussion with treasury using our language. This is about building a bridge and then translating in terms they understand and then it will be useful.

PVMF peer review - Working Group meeting Minutes 01/03/12 page 1 of 4




FILE NO: 11/135

SUBJECT: PVMF peer review - Working Group

DATE: 1 March 2012, 10am

ATTENDEES: Jacqui Allen, Marty Cunningham, Oliver Gatty, Ricky Arnold, Rebecca Sheardown (minutes)

Michael Chappell (arrived 10.25am), John Knell

APOLOGIES: Colin Walker, Chris Penwald

Penny Knight – is no longer working at Dept Finance


1. Introduction

JA provided an overview of the meetings during the week for those present who were not in attendance at any of the other sessions.

2. Summary of Peer Review sessions

JA gave a summary of all meetings • Executive group – There was a positive response

from key business areas and acknowledgment that there is an impact across DCA as a portfolio and decisions to be made.*

• DSD Staff and DCA Managers – More positive response and an understanding on how the metrics can be applied across programs. Some formal feedback has been received and forwarded to Pracsys by email from MC.

JK – Focus Groups feedback • Found the organisations were comfortable overall

with the concepts and definitions being delivered. • Understanding that if the model works that it will

change the common language of measuring within the sector.

• Organisations understood the value of data DCA is asking for from them and how it supports DCA’s case to Government.

MCh – Focus Groups feedback • There appeared to be less challenges than the

previous testing workshops in Dec 12. This shows that most of the changes were addressed by those panellists and that it is possible the definitions are

DCA Executive has portfolio decisions to make regarding the impact of the PVMF.


settling into the final list.

3. Intellectual Property

JA understands that the IP is in specifically in Form A & B (definitions) but not in the background work. MCh states that Alec Coles from Museum is keen to meet with consultants regarding the work. JA clearly states that there is not to be any intellectual property information divulged to MuseumWA. Portfolio meetings are to be dealt with strategically by DCA.* Intellectual Property – MCh understands that the definitions of Quality and Reach and the cultural infrastructure systems are specific to DCA in the logic framework, but that the rest of the model is based on international public value concepts. JK – without the DCA model and DCA measures there is not much that framework presents and would only prompt interested parties to contact DCA for more information. Final report – MCH agrees there should be an example provided to demonstrate how the measures are collected through the framework.*

Pracsys are not to disseminate any intellectual property information to MuseumWA. Portfolio meetings are to be dealt with strategically by DCA.* Pracsys to provide an example to demonstrate how the measures are collected through the framework in the Final Report

4. Development of PVMF Final Report

JK - States that the prompts and development of the measures has been developed intellectually and the formula as such has not been documented. JA – Would like the Final Report to contain a guide as to how to use the model when we are developing the implementation of the new system.* MC – agrees that the definitions of the meta-data are required. MCh - suggests that even a small testing series could be done to gather data by using an iPhone App (as discussed in previous meetings). MC – the richness of the development needs to be reflected in the Final Report. Responses from draft report has been finalised and is awaiting final signoff from Jacqui which will then be presented to Pracsys.* MCh – The acceptance of the Final Report which is not just what’s in the RFQ but also in the Pracsys proposal (RFQ content). It is suggested that the final report also includes all diagrams, the measurement database and also all minutes from meetings as appendices.* JK – Identifies that economic value needs to be more explicit in the final report.* JA – Regarding the length of the report, Jacqui states that a shorter report with a lot of appendices would be better. The report also highlights improvements that can be made in regards to policy for creative industries.* The report needs to be in in the third person style. JK – The feedback received by email from Dan Minchin (WAM Chair) shows very clearly that he sees the model delivers cultural measurements and the economic

Pracsys to contain a guide as to how to use the model when we are developing the implementation of the new system in the Final Report. DCA to approve ‘Responses from Draft Report’ and send to Pracsys. Pracsys to include all changes identified through the meeting for inclusion in the Final Report. Pracsys to clarify where in the reporting process the measures are collected and how it feeds into the RSP.

MCh to design


measure need to be more explicit JA – Wants to clarify that reporting from Organisations and applicants will still require financial and business reporting and that the PVMF is imbedded in the financial data processes. MCh and JK both agree this is where is should be recorded and will make it clearer in the final report and feeds into the DCA Recommendation Scoping Project (RSP). JK – Suggests that a ’dashboard’ analysis assist with making key decisions and suggests asking for receipts is essential for clarifying and organisations leverage. JA – confirms that it would be important for this level of reporting. MCh - Has an idea on how to capture all the 15 levels of measurement and the next layer down in a diagram and will add this to the final report.* MCh - Will also clarify how our state structure differs from other agencies to clarify the terms of infrastructure complexities and accountabilities.* JK - Sees that the logic framework can deliver a “Creating Investment Policy” under the cultural infrastructure ecology and it could then be pushed out to the sector and almost becomes an investment prospectus. JK - Confirms they are meeting with David Throsby on Friday 2 March and then Queensland University of Technology the following week. Final Report writing style – discussion concluded that the writing style would be edited for a corporate audience.

diagram to represent the 15 levels of measurement.

Pracsys to clarify the differences in other state agencies in terms of infrastructure and accountabilities.

Pracsys to edit final report writing style for a corporate audience.

5. Australia Council seminar ‘Culture Count: Measuring Cultural Value’ 20 March 2012

Australia Council seminar ‘Culture Count: Measuring Cultural Value’ 20 March 2012 • JK knows Hasan Bakhshi (NESTA) and understands

the policy work being done is the early stages of investigation and no paper is being provide, only panel discussion.

• This work is an opportunity for DCA to be part of the public value debate and suggests it could be either Michael Chappell or a DCA rep on the panel.

• JK does have a public version of the logic framework stripped of any DCA information.

• The risk is that there is industry debate about DCA’s framework ahead of any formal public briefing paper.

• JA wants to see and approve this public version of the model and have the Ministers approval before it is distributed. MCh agrees to provide the Briefing Paper to JA for approval.*

Pracsys to provide DCA with a Briefing Paper for Ministerial approval.

6. Weightings Weightings - Roundtable discussions were had to clarify how the Weightings would be addressed. • Result is that Weightings would be determined under

each funding program and they will be determined in agreement with the Organisations.

• An example should be provided in the Final Report.*

Pracsys to demonstrate a use of weightings by programs and provide an example in the


Final Report.

7. Key Performance Indicators

Key Performance Indicators • Decisions will need to be made by DCA as to how we

will report KPI’s either measure by measure or aggregated up to one number.*

• Suggestion it could be divided into the four areas of Quality, Reach, Impact and Value.

• OG suggested that you could provide Government with one number and then the narrative and evidence is contained within the annual report that shows how the final figure is determined.

DCA needs to decide on KPI reporting scales after PVMF is finalised.

8. Definitions • The Definitions cannot be changed over time and

prompt questions are required to get the response in the data being collected.

• DCA will need to work on these prompt questions during implementation.

• Examples can be found on http://www.khanacademy.org/ (more information required from JK about which section on website)

9. Academic review • MCh and JK presume there will not be any adverse

reaction to the DCA work as they will only present the three diagrams; Creating an operable measurement framework, Public Value Measurement Framework Process Diagram and Creating a learning system

• JK and MCh are to provide feedback after all meetings held over east.*

• The review will focus on the context and scope of the project.

• MCh and JK will ask the reviewers to consider if the process has been rigorous within a public funding context.

Pracsys to provide reports after all eastern states meetings.

10. Implementation • MCh advises that the framework would be best

implemented in small parts instead of a major overhaul.

DCA is responsible for implementation of the PVMF.

11. Communication • JA confirms that a public message is needed for the

sector to continue ongoing consultations and verification the Creating Value policy is being fulfilled.*

DCA develop Communication Plan for public message.

12. Conclusion Confirming actions during the meeting • Responses from the Draft Report to be approved by

JA and sent to Pracsys. • Minutes from all meetings to be approved and sent to

Pracsys. • MC and RS to meet with Pracsys to clarify

appendices to be included in Final Report. Pracsys estimate Final Report two weeks following this date.

• Pracsys to provide reports from all eastern states meetings to be provided to DCA.

• Pracsys to provide a Briefing Paper that is their public version of the model that will sent for Ministers approval.

DCA - MC and RS to meet with Pracsys to clarify appendices to be included in Final Report.

57

A6

APPENDIX 6: FORM A B

58


Applicant:

Application Scope:

Dimension Parameter Definition Funding Self Assessment (A1) Comments Funding Panel Assessment (A2) Comments

Quality Creativity Inquisitiveness The extent to which the work promotes curiosity in artist and audience

Imagination The extent to which the work explores new possibilities or views

Originality The extent to which the work breaks new ground (modes of practice or content *)

Risk The extent to which the artist is fearless and negotiates new artistic approaches

RigourThe extent to which the work has undergone thorough research and development

Currency The timeliness of creative idea in relation to contemporary events

Authenticity The extent to which the work respects cultural tradition or is unique to WA

Innovation The work demonstrates an ability to realise creative ideas to real world outcomes

Excellence The work is widely regarded as best of its type in the world

Reach Audience Number Number of people in communities of interest who directly engage with the work

Diversity The extent to which the work engages a broad cross section of society

Connection The quality of the connection of the work with communities of interest

Practice Collaboration The extent to which the work connects with communities of practice

Leverage The ability to attract investment from a range of non- DCA sources

PlatformThe capacity of the work to have long term influence and importance to communities of interest and practice

Glossary

Community of interest is the section of the public who engage directly or indirectly with the work; or who have similar characteristics and attitudes to those who do engage.Community of practice includes the artists and collaborators directly involved with the work; or who have similar characteristics and attitudes to those who do engage.

Applicant:

Application Scope:

Dimension Parameter Definition Acquittal Self Assessment (B1) Comments Acquittal Panel Assessment (B2) Comments Public'Assessment'(B3) Comments

Quality Creativity Inquisitiveness The extent to which the work promotes curiosity in artist and audience

Imagination The extent to which the work explores new possibilities or views

Originality The extent to which the work breaks new ground (modes of practice or content *)

Risk The extent to which the artist is fearless and negotiates new artistic approaches

RigourThe extent to which the work has undergone thorough research and development

Currency The timeliness of creative idea in relation to contemporary events

Authenticity The extent to which the work respects cultural tradition or is unique to WA

Innovation The work demonstrates an ability to realise creative ideas to real world outcomes

Excellence The work is widely regarded as best of its type in the world

Reach Audience Number Number of people in communities of interest who directly engage with the work

Diversity The extent to which the work engages a broad cross section of society

Connection The quality of the connection of the work with communities of interest

Practice Collaboration The extent to which the work connects with communities of practice

Leverage The ability to attract investment from a range of non- DCA sources

PlatformThe capacity of the work to have long term influence and importance to communities of interest and practice

Glossary

Community of interest is the section of the public who engage directly or indirectly with the work; or who have similar characteristics and attitudes to those who do engage.Community of practice includes the artists and collaborators directly involved with the work; or who have similar characteristics and attitudes to those who do engage.

59

A7

APPENDIX 7: ISSUES PAPER 1 AND 2

DRAFT



ISSUES PAPER 1 - REFERENCE REVIEW

AUGUST 2011

DRAFTDISCLAIMER



Document Control


v 1.0 Issues Paper 1 Jenny Nichol Michael Chappell 23 August 2011

23 Lyall Street, South PerthWestern Australia, 6151t (08) 9367 1511f (08) 9367 [email protected]

i

DRAFTCONTENTS

1 INTRODUCTION 1

1.1 Project Context 1

1.2 Issues Paper 1 Purpose 1

1.3 Public Value Management Framework 1

2 STRATEGIC CONTEXT 3

2.1 Whole of Government 3

2.2 Culture and the Arts 3

2.3 A creative, sustainable and accessible culture and arts sector 3

2.4 Western Australia’s natural, cultural and documentary collections are preserved, accessible and sustainable 4

2.5 KPI Review 4

2.6 DCA Policy and Objectives 5

2.7 Funding Process Review 5

3 REFERENCES REVIEW 6

3.1 Reference Summaries 6

3.2 Analysis 10

3.3 Conclusion 11

4 PERFORMANCE ASSESSMENT FRAMEWORK 12

4.1 Logic Framework 12

4.2 Inputs 13

4.3 Process 13

4.4 Outputs 13

4.5 Outcomes 13

4.6 Impact 13

4.7 Value 14

APPENDIX 1 15

APPENDIX 2 17

ii

DCA - Issues Paper 1: Reference Review

DRAFT

1

1.0DRAFT1 INTRODUCTION

1.1 PROJECT CONTEXT

The Department for Culture and the Arts (DCA) is undertaking a ground-breaking initiative to understand and measure the public value it creates through its role in Western Australia. This project aims to be holistic in not just creating a measurement framework but in creating distinct links back to policy and funding decisions so that the framework is a dynamic informer of an evolving public value contribution. This should allow for the WA public to be assured of the value of their investment in WA’s cultural capacity and at the same time experience more direct benefit from an expanding and dynamic cultural offer.

To those ends, this project aims to create an overall measurement framework that simplifies the task for the DCA to shape its portfolio of investments to ensure they produce the impacts and public value outcomes it wishes to prioritise. The aim of this project is to create a family of metrics that connect the DCA’s defined outcome areas (people, communities, economies and environment) with the DCA’s stated policy drivers - quality, reach, impact and value. This set of metrics will be called the Public Value Measurement Framework (PVMF).

1.2 ISSUES PAPER 1 PURPOSE

The purpose of Issues Paper 1 is to summarise the current state of value measurement at Department of Culture and the Arts; and to identify some of the gaps to be addressed in the new performance management framework.

1.3 PUBLIC VALUE MANAGEMENT FRAMEWORK

The objective of the Public Value Measurement Framework (PVMF) is to identify a methodological framework to provide workable data in terms of measures and indicators, including methods of data collection, of the following concepts that can be used to evaluate the range of public value outcomes delivered by the Department and its clients.

Quality: Quality is related to the creative process and product and will include the distinctive, innovative and significant elements of the creative experience. Dependent on context, it is anticipated that quality will be measured by a combination of self assessment, peer assessment, audience and public engagement and response.

Reach: Reach is seen the extent of access to and participation in arts and cultural activities. DCA seeks to measure the breadth and depth of engagement through attendance and participation data alongside audience and public satisfaction with their level of engagement.

Impact: Impact is seen as the social, cultural and economic effect of engagement in arts and culture activities and will include the transforming impact through engagement including the public as active creators of artistic and cultural work. Impact will be measured through the quantitative and qualitative review of outcomes from engagement with arts and cultural activities.

2


DRAFTValue: Value in this context is seen as the economic value of arts and culture and the appreciation of arts and culture in the WA community. Value in this context can include analyses of Contingent Value, Willingness to Pay and other concepts that help demonstrate the value people place on arts and cultural activities.

DCA is aware that there is limited work in this area and therefore this project aims to explain its assumptions for the measures and indicators provided including working definitions of these four concepts. The PVMF will provide recommendations for the data to be collected including meta data for these four concepts.

3

DRAFT2.0

Performance measurement is a consistent thread through governments. The usefulness of indicators should be judged by the degree to which they are used to make strategic and operational decisions – particularly in the allocation of resources to achieve desired outcomes.

2.1 WHOLE OF GOVERNMENT

The Deportment of Culture and the Arts contributes to the achievement of the Government goal: “Greater focus on achieving results in key service delivery areas for the benefit of all Western Australians”.

2.2 CULTURE AND THE ARTS

The Government’s Desired Outcomes for the culture and arts portfolio are:

• A creative, sustainable and accessible culture and arts sector

• Western Australia’s natural, cultural and documentary collections are preserved, accessible and sustainable

Inherent in these Outcomes will be a creative community that participates in and values culture and the arts, contributing to the quality of life and well-being of all people throughout WA. The Key Performance Indicators (KPIs) supporting are intended to provide the evidence that the resource allocation decisions made by the Department of Culture and the Arts are driving activity towards the Outcomes.

2.3 A CREATIVE, SUSTAINABLE AND ACCESSIBLE CULTURE AND ARTS SECTOR

This outcome has three effectiveness indicators. The first of these is Creativity.

Creativity is measured as the proportion of funding applicants who are satisfied with the key elements of the creative funding programs (biennial Client Satisfaction Survey). Creativity is said to ‘depend on the cultural values, preferences and realities of residents and other stakeholders in a given community’. The resource allocation decisions are made in each of the funding programs by peer panels. The target for this measure is 79%.

The second Key Effectiveness Indicator is Sustainability.

Sustainability is measured in two ways: 1) by the perceived value of the culture and arts to the WA community and 2) the proportion of triennially funded organisations within the culture and arts sector which are regarded as financially healthy. The target for these two measures are 76% and 20% respectively. The first measure provides an assessment of the extent to which the community values the culture and arts sector (measured through the Arts Monitor). The second provides an indication of the financial health of the triennially funded organisations that make up a significant proportion of WA’s subsidised culture and arts sector.

2 STRATEGIC CONTEXT

4


DRAFTThe third Key Effectiveness Indicator is Accessibility, which is measured as the proportion of West Australian’s that perceive culture and arts events to be accessible (gathered through the Arts Monitor). The target is 51%.

The Key Efficiency Indicators for the portfolio relate to cost per grants and cost per performance.

2.4 WESTERN AUSTRALIA’S NATURAL, CULTURAL AND DOCUMENTARY COLLECTIONS ARE PRESERVED, ACCESSIBLE AND SUSTAINABLE

The first Key Effectiveness Indicator is the extent to which WA’s natural, cultural and documentary collections that require preservation are preserved (as a percentage). This provides an assessment of the extent to which the State’s art, museum, heritage and archives collections are preserved. It is explained that since preservation of the entire collection is not required, this measure only relates to that part of the collection that is determined by the organisations as requiring preservation. So the indicator measures the extent to which the portfolio’s collection preservation outcomes are met.

The second Key Effectiveness Indicator relates to accessibility and is assessed based on 1) the number of accesses to WA’s natural, cultural and documentary collections per capita; 2) percentage of clients satisfied with the services associated with accessing WA’s natural, cultural and documentary collections and 3) number of accesses (attendance figures at Museum, Art

Gallery, State Library and State Records office and accesses via Internet User sessions and material exchanges between State Library and Local Government Libraries. The targets are 2.224, 88% and 4,823,000 respectively.

The third Key Effectiveness Indicator relates to sustainability. This is the value of collection renewal content development, expansion and or maintaining the physical integrity of the state’s art, museum, heritage and archives collection as a proportion of collection value. The target is 6%.

The Key Efficiency Indicators for the second Government Outcome all relate to cost of service delivery/ access for the Art Gallery, State Library, Museum, Government Record keeping service (ie relating the resource input to the services provided).

2.5 KPI REVIEW

The DCA’s KPIs are currently under review. The issues relate to be address by the review are:

1. The existing creativity measures do not accurately measure creativity.

2. Key effectiveness indicator Accessibility is based on community perceptions measured through the annual Arts Monitor survey but do not directly relate to DCA core processes and, therefore DCA is unable to directly influence the measures through changes in performance.

3. The key efficiency indicator for Arts Industry Support is not an accurate reflection of efficiency and is based on historical calculation assumptions that may no longer be valid.

5


DRAFT4. There is limited, if any, use of current KPIs

in the day-to-day management of the DCA.

2.6 DCA POLICY AND OBJECTIVES

In 2010 the Department of Culture and the Arts (DCA) released two new guiding documents, its 2010-2014 Strategic Plan and the 2010-2014 Creating Value; an Arts and Cultural Sector Policy Framework. Each document emphasises the delivery of public value as the driving principle of the work of the Department and by extension the outcomes of its investments.

2.7 FUNDING PROCESS REVIEW

The Funding Process Review aims to assess how DCA can maximise the public value delivered by its funding activities and programs. The Development and Strategy Directorate (DSD) works with a range of stakeholders to provide support through the development of arts and cultural policy; research, distribution of funding and implementation of State and national strategic arts and cultural programs. Within this, DSD provides funding support to the arts and cultural sector through recurrent funding to 49 key organisations and offers grants programs across a range of arts and culture sectors.

How DSD aims to deliver Public Value to the Western Australian community through the delivery of services and support for arts and culture is outlined in Creating Value; an Arts and Culture Sector Policy Framework.

Creating Value outlines the framework for delivering public value through the:

Priority principles of:

• Creativity

• Engagement

Delivering outcomes in the areas of:

• Creative People

• Creative Communities

• Creative Economies

• Creative Environments

Utilising the following strategies:

• Funding

• Partnerships

• Infrastructure Support

• Sector Development

6

DRAFT3.0

3.1 REFERENCE SUMMARIES

We reviewed a range of references provided by DCA staff and otherwise sourced. These are summarised below, and categorised by type in Appendix 1.

3 REFERENCES REVIEW

Funding Process Review – Stage 1

DCA

This review looked at how funding is allocated and how to improve the funding process (in relation to staffing, online access, panel selection, funding application, review and acquittal process). It identified a lack of application of acquittal information against DCA objectives.

Acquittal Data Review Project Report on the Data Collection Review

DCA & ABS

Acquittal data are collected to assess the projects funded by DCA to ensure they meet public funding requirements. The review identified a lack of consistency of data collected across programs making aggregation difficult. The review also cited poorly defined data needs and a requirement to redesign application and acquittal forms with more relevant and targeted questions.

Department of Culture and the Arts DSD Client Satisfaction Survey 2010

Patterson Market Research

DCA’s annual survey of key funded organisations, Grant recipients and Non-successful applications. Survey questions cover satisfaction with DCA by Clients and can be broken down by art form (eg: dance, music, visual arts etc.) Satisfaction includes whether DCA activities cover needs of government, arts sector or general community. DCA processes, relationship with DCA, reporting and acquittal, DCA Communications, publications, website, policy and strategy.

Arts Monitor Report Final Report 2011

Patterson Market Research

DCA’s annual survey of the general public about their cultural activity. It covers arts and cultural attendance (what) and why, expenditure, source of information to find out about cultural events, how they book and how valuable arts and culture is in their life.

Review of DCA Key Performance Indicators 2010

DCA

This outlines the internal project to review DCA’s KPI’s. The issues to be reviewed:

1. Existing creativity indicator measures do not accurately measure creativity

2. Key effectiveness indicator of measure isn’t actually able to be influenced by DCA

3. Others including lack of use of KPI’s in day to day management

Development and Strategy Directorate (DSD)

DCA

DCA’s Development and Strategy Directorate’s KPI’s and how they are calculated:

• Creativity

• Sustainability

• Accessibility

• Service: Arts Industry Support (cost/grant, total cost of services, number of grants)

7


DRAFTDCA Project Management Framework

DCA

Minimum requirements for the management of projects within DCA.

Arts Organisations Financial Data Dictionary

Australia Council for the Arts

Financial data currently sought by Australia Council from funded organisations (gathered electronically).

Harmonised Reporting Framework – Review of national arts organisations statistical data set


A national arts organisation statistical data set that is part of the harmonised reporting framework. A review of the data set has been conducted which identified some gaps. Major performing arts organisation must submit their financial data to Australia Council on a quarterly basis via an online system. Productivity commission has developed a framework that defines output categories enabling the ability to measure outcomes and impacts. This framework helps give context to statistical data collection.

Artistic Vibrancy: Self-reflection Tool


A framework/set of tools to help major performing arts companies to achieve “Artistic Vibrancy” ie. best practice in arts management.

Defining artistic vibrancy: A discussion paper for the major performing arts sector, Australia


A discussion paper on what defines an artistically vibrant arts company.

Meaning measurement: A review of the literature about measuring artistic vibrancy


A review of the literature about measuring artistic vibrancy identifies a range of research papers trying to assist in measuring performance in the arts. P15 discusses Bovaird who measures effectiveness = throughput x average impact. Throughput = numbers of people (ie. Audience, target group) Impact = changes in client as a result of experience (happiness, enjoyment, pride, cultural exchange, client satisfaction, repeat visitors, willingness to pay). Also discusses role of funding agencies and what they seek for reporting. Need to seek measures of artistic quality as well as financial measures.

“Tell me Honestly…”: Good practice case studies of artistic self-assessment in performing arts organisations


Case studies from successful major performing arts organization about how they monitor/measure and improve their business to achieve best practice, vibrant arts outcomes.

Vital Signs Cultural Indicators for Australia

Cultural Ministers Council

A very broad summary of data available that could be used as measures of cultural industry impact in Australia. The metadata covers three broad categories of measurement:

1. Economic development

2. Cultural Value

3. Engagement and Social Impact

8


DRAFTPublic Value International Benchmarking Research Project for Department of Culture and the Arts 2010

Luke Fussell, UWA Student Placement

A survey of international case studies of attempts to evaluate the arts. It surveys Arts Council Ohio, Arizona Commission on the Arts, Arts Council England, BBC. BBC came up with 4 main drivers of public value: Reach, Value for money, Quality, Impact.

Culture Report 2010 Western Australia, National Centre for Culture and Recreation Statistics

Australian Bureau of Statistics

A comprehensive summary of ABS data relating to the cultural sector in WA covering:

• Attendance and participation

• Employment, work, volunteers

• Children and youth

• Indigenous participation

• Overseas-born participation

• Disabilities participation

• People of age 60 participation

• Household expenditure

• Economic activity

• Cultural funding by Government and business

• Museums, Art Galleries, Libraries, Performing Arts, Cinema, Film, Video

• Data Sources about culture

Statistical publications and feature articles information papers

Australian Bureau of Statistics

Analysis of:

• Participation, Involvement, Employment and Attendance

• Expenditure

• Museums, Art Galleries and Libraries

• Literature and Print Media

• Music and the Performing Arts

• Broadcasting, Electronic Media and Film

• Built and Environmental Heritage

Contribution of the Not for Profit Sector, Productivity Commission

Productivity Commission

A discussion on how to measure the contribution of the not for profit sector. It breaks the process of the production value chain of the NFP sector into five stages: Inputs, Activities, Outputs, Outcomes, Impacts. Four of these can be used in the measurement of effectiveness of the sector: Inputs, Outputs, Outcomes and Impacts. Inputs and outputs are relatively easily quantified. Outcomes and impacts are vital as this is where NFP sector makes a difference, but are harder to measure. P10 Appendix B shows a framework for measuring outcomes in the performing arts.

9


DRAFTArts Funding, Austerity and the Big Society: Remaking the case for the arts

Arts Council England

This document argues the case for finding measures to value the arts.

Grant Acquittal Form 2010

DCA

The form used by grant recipients to acquit their grant. Information collected includes qualitative information on project aims and results plus financial information, participation numbers, employees, media coverage, diversity of involvement by different community groups and feedback of quality of service by DCA.

Looking Forward Fund Acquittal Form

Acquittal form for looking forward fund. Similar information collected as for other grants, plus information about amount of leverage through sponsorship.

The Culture/Economy Balance

Creative Consultancy

Discussion of the opportunities for arts spaces to engage with audiences through new media, including giving audiences increasing engagement/involvement in creation of the arts.

Connecting: Arts Audiences Online


A practical guide for how arts groups can engage with their audiences through using new media to improve marketing exposure and involvement of audiences.

More than bums on seats: Australian participation in the arts


A research paper that found that Australians are actively involved in the arts and opportunities to use the internet to involve people in the arts exist and are increasingly being used.

Measuring the value of culture: a report to the Department for Culture Media and Sport

Department for Culture, Media and Sport, Economic and Social Research Council and Arts and Humanities Research Council

This document is quite a thorough review of methods for the cultural sector to demonstrate its benefits. It searches for methods that fit the British governments framework, which requires Cost Benefit Analysis (CBA) to justify government decisions. It argues that stated preference methods such as contingent valuation should be used for decision making about cultural policy. Choice modeling is another preferred method.

10


DRAFT3.2 ANALYSIS

The references can be analysed in four categories.

3.2.1 Goals and Strategies

The first category includes documents dealing with DCA’s goals and strategies. Ultimately the PVMF must be a mechanism which feeds information back to the DCA about performance of its programs against the measures outlined in the documents.

3.2.2 Process

The second category relates to Process. These documents describe the ways that DCA allocates its resources to achieve programs goals. The outcomes of the PVMF will need to align with the review of the acquittal process to ensure that information collected during future acquittals can feed directly into the measurement framework.

3.2.3 Data

The third category relates to Data. These references detail the type of data that is or could be used by the PVMF to provide feedback of the cultural sector’s performance against the key metrics of Quality, Reach, Value and Impact. Our current understanding of this data is presented in Appendix 2, which describes the availability of data which can provide measurable insight into the performance of DCA’s funded organisations and grant programs. This data set will form the basis for the identification of new measures necessary to capture the complete picture of DCA’s programs’ performance against the Quality, Reach, Value and Impact measures.

3.2.4 Theory

The fourth category lists information that was reviewed relating to the theory and research of measuring the value of the arts.

3.2.5 Data Available

A wide range of data are available which quantify the Inputs and Outputs from the sector. The data from the acquittal forms and harmonized reporting framework for major performing arts companies provide a vast range of figures on spend, employment and income. These provide baseline information which will be used to evaluate the value of government’s investment once other measures are identified.

In addition information is relatively available which relates to Reach, including information relating to number of performances, exhibitions etc, diversity of audiences, diversity of locations, philanthropy and sponsorship. This data can be valuable in contributing to an understanding of the reach of DCA”s funded programs.

There is WA-specific meta-data available which provides information on cultural participation (Culture Report, ABS and Arts Monitor) as well as expenditure, barriers to attendance and information sources about cultural activity. The Arts Monitor also reports on attitudes to arts and culture and accessibility of the arts. The ABS data can also tell of the average time spent on culture and leisure activities; economic activity, employment statistics; children, indigenous, people with disabilities, older people and overseas born participation in the arts.

11


DRAFTThe Vital Signs Cultural Indicators document proposes a rang e of documentation that be useful in measuring contribution of arts and culture to the public good. The first section identifies ‘economic development’ measures and once again these are really just output measures (ie number of people employed, household expenditure, visitor expenditure, government support for culture). There is meta-data listed from Australia Business Arts Foundation relating to private sector support for culture, which could be used to evaluate Reach and information on Volunteer work in arts and culture which may be a measure of impact. A measure of economic contribution of cultural industries could be a measure of value.

The second section identifies ‘cultural value’ indicators. The number of cultural assets across Australia is said to indicate the size and diversity of the nations material cultural heritage. It also argues that the number of grants, tax arrangements, programs, training, fellowships etc can indicate the development and support for the arts. As argued in this paper, these are primarily input measures but could demonstrate Reach. There is an attempt to measure innovation through the number of projects funded through funding programs, arts incubators and subsidies. This is a measure of inputs once again and can help is assessing value. Touring data is useful to measure reach The third section identifies ‘ engagement and social impact’ as a measure of cultural value. Attendance and participation numbers, numbers of regional programs and number of students in arts education are provided as indicators of this measure. Once again these are inputs and could demonstrate reach but do not contribute to quality assessment.

Along with some useful measures of Reach, this meta-data is helpful in providing some insight over time about attitudes of West Australians to the cultural sector although it doesn’t provide the DCA with any direct feedback relating to specific expenditure through its Key Funded organisations nor its grants programs. Some of the data can assist DCA to measure state-wide impacts and there is a place for this meta data in assisting the DCA in evaluating attitudes and impacts over time to inform policy decisions. The data should be used at the portfolio level to inform policy and funding allocations over time.

3.3 CONCLUSION

The review demonstrates quite clearly that this project being undertaken by DCA is groundbreaking since no comprehensive system of measurement has been successfully established. In particular, the research reveals that clear definitions have not been established for Quality, Reach, Value and Impact nor has any organisation successfully established a framework to measure them.

12

DRAFT4.0

4.1 LOGIC FRAMEWORK

The following provides initial definitions of the key measurement concepts. Figure 1 is a logic framework that places the concepts in the context of their role in the strategic resource allocation process.

4 PERFORMANCE ASSESSMENT FRAMEWORK

Figure 1: Logic Framework Diagram

Source: Pracsys 2011; Productivity Commission, 2011

13


DRAFT4.2 INPUTS

Inputs are the operational resources that the he arts/cultural entity uses to undertake its work. Resources are easily accounted for and often consist of easily quantifiable sets of information. Many inputs can be used to provide an understanding of baseline data which when divided into impact measures, will provide a way of measuring value.

4.3 PROCESS

The Process is the activities the means by which inputs are converted into outputs in order fro the arts/cultural entity to achieve its mission.

4.4 OUTPUTS

Outputs are the direct products of the arts/cultural entity. Outputs are not ends in themselves but are a means of delivering outcomes. Outputs are also usually easily quantifiable. These can be used to also assist in evaluating value and must be used as the basis for developing metrics which will be used to evaluate Quality and Reach.

4.5 OUTCOMES

Outcomes are the intended and unintended results of the outputs. While output measures may usually place a dollar value on outputs, outcome measures go a step further by capturing the full benefit over and above market price for the activity. It is the identification of outcomes that will highlight the achievement of Quality and Reach measures. Some preliminary definitions of the keys concepts follow.

4.5.1 Quality

• Creativity is the ability to make something new through the use of imaginative skill

• Innovation is the process of applying creativity to a problem, issue or need felt by a community of interest

• Entrepreneurship is the process of gathering scarce resources to make the process of innovation repeatable and sustainable

• Capacity is the ability to repeat the innovation process with new creative input underpinned by the entrepreneurial resources

4.5.2 Reach

• Audience numbers, location (region, metro, national, international), diversity and seasonality

• Engagement type (product, process, experience) – including online

• Artist leverage (networks)

• Corporate leverage

• Philanthropic leverage

4.6 IMPACT

According to the Productivity Commission’s model, impacts are the long-term effects produced by an activity. Impacts, as distinguished from outcomes, capture the long term and feedback effects and spillovers from outcomes. As mentioned elsewhere, impact could be measured as the result of quality outcomes multiplied by reach outcomes.

14


DRAFTThe following parameters could be used to distinguish direct impact results:

• Communities of interest impacts

• Communities of practice impacts

• Duration of impact

The following parameters could be used to distinguish indirect impact results:

• Opportunity cost

• Contingent valuation (fear of withdrawal)

4.7 VALUE

Definitions of value are widely debated, however as outlined in the Creating Value Policy Framework, value in this context is seen as the economic value of arts and culture and the appreciation of arts and culture in the WA community. In this context value could be measured by dividing the impacts by the inputs.

Value has a range of parameters:

• Community of interest

• Community of practice

• State, national and international

15

DRAFTA1

APPENDIX 1

17

DRAFTA2

APPENDIX 2

Performance Dimension Parameter Definition Metric Unit of Measure Data Type Data Source Document

Reference #Data Collection

Data Analysis

Decision Rule

Indicator Strength

Quality Creativity Creativity is the ability to make something new through the use of imaginative skill Newness, originality imaginative skill Likert scale Peer review peers none weak

InnovationInnovation is the process of appying creativity to a problem, issue or need felt by a community of interest

Define audience needs and wants, output addresses needs and wants

targeted enagagement Likert scale Arts Monitor public none weak

EntrepreneurshipEntrepreneurship is the process of gathering scarce resources to make the process of innovation repreatable and sustainable

Resources assembled (all sources) $ (all sources), network connections $, people Grant aquittal self-grantee none weak

CapacityCapacity is the ability to repeat the innovation process with new creative input underpinned by the entrepreneurial resources

Reach Audience locationNo of performances, productions, exhibitions, publications, sales, tours, acitivties by metro, regional, inter, intrastate

numberMPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

Audience diversity attendance number by regional, metro, interstate, international and target area

number-paid unpaid and total

MPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

attendance number at non-ticketed activities by metro, regionial, interstate, target area, international

numberMPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

participant numbers (paying, non-paying) by metro, regional, interstate, target area, international

number-paid unpaid and total


8 self-MPA

Audience number attendance number number-paid unpaid and total


8 self-MPA

Broadcast audience number

2011 Harmonised Reporting Templates-Quantitative Data-Glossery of Terms

8 self-MPA

paid and unpaid attendance by perofrmaces, exhibitions, residences, forums, festivals, creative developments

number paid and unpaid

Looking Forward Fund Acquittal Form 17 self-grantee

Audience seasonalityCorporate Leverage Sponsorship cash and inkind $ MPA's financial reporting 8 self-MPA

sponsorship, funding and DCA grants $ Looking Forward Fund Acquittal Form 17 self-grantee

Sponsorship and inkind $ Contemporary Music Acquittal form 16d self-grantee

Sponsorship and inkind $ Arts Grants Acquittal form 16c self-grantee

Sponsorship and inkind $ Designer Fashion Acquittal form 16b self-grantee

Philanthropy Leverage Donations $ MPA's financial reporting 8 self-MPA

$ Looking Forward Fund Acquittal Form 17 self-grantee

Foundations $ MPA's financial reporting 8 self-MPAFundrasiing $ MPA's financial reporting 8 self-MPA

Membership/Subcribers

Financial and non-financial members by metro, regional, state, interstate, target area, international

number- financial, non, total


8 self-MPA

subscribers totalMPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

Social Change?Diversity of delivery mechanism

LocationsNumber of activities, performances, exhibitions, residences, fesitvals, publications by local, regaionl, national international

number MPA's financial reporting 8 self-MPA

Impact-Direct Communities of interest Number of artists supported/represented number


8 self-MPA

Number of particpants in the activity number Looking Forward Fund Acquittal Form 17 self-grantee

Media coverage by news, magazine, radio, TV number Looking Forward Fund Acquittal Form 17 self-grantee

satisfaction

Performance Dimension Parameter Definition Metric Unit of Measure Data Type Data Source Document

Reference #Data Collection

Data Analysis

Decision Rule

Indicator Strength

value for moneyripple effect

engagement Responses to information requests numberMPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

non-artists supported numberMPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

volunteers number and hoursMPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

unpaid personnel involved in project number Looking Forward Fund Acquittal Form 17 self-grantee

Impact-Indirect Opportunity costFear of withdrawalContingent valuationPerth/regions/WA

Value IndividualCommunityState

Inputs Government subsidy Grant/subsidy funding income $ MPA's financial reporting 8 self-MPATotal cost Total expenditure $ MPA's financial reporting 8 self-MPA

Total expenditure $ Looking Forward Fund Acquittal Form 17 self-grantee

Total expenditure $ Contemporary Music Acquittal form 16d self-grantee

Total expenditure $ Arts Grants Acquittal form 16c self-grantee

Total expenditure $ Designer Fashion Acquittal form 16b self-grantee

Employee numbers number of full-time, part-time and casual numberMPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

number of full-time, part-time and casual number Looking Forward Fund Acquittal Form 17 self-grantee

Employee costs Salaries, wages, fees $ MPA's financial reporting 8 self-MPA

Salaries, fees, living allowances $ Looking Forward Fund Acquittal Form 17 self-grantee

Production Costs Production /Exhibition/Touring expenses $ MPA's financial reporting 8 self-MPA

Production /Exhibition/Touring expenses $ Looking Forward Fund Acquittal Form 17 self-grantee

Other programs costsWorkshops, Development, Education,, Publications, Cost of sales, Evaluation and Research, Community programs,

$ MPA's financial reporting 8 self-MPA

Outputs Sales

Income earned, inkind, sponsorship and total $ Looking Forward Fund Acquittal Form 17 self-grantee

income from ticket sales, product sales $ Looking Forward Fund Acquittal Form 17 self-grantee

income, sponsorship, inkind $ Contemporary Music Acquittal form 16d self-grantee

income, sponsorship, inkind $ Arts Grants Acquittal form 16c self-grantee

income, sponsorship, inkind $ Designer Fashion Acquittal form 16b self-grantee

Profit/Loss Profit/Loss $ Looking Forward Fund Acquittal Form 17 self-grantee

Profit/Loss $MPA's financial reporting- Quantitative Data Summary 2010

8 self-MPA

Profit/Loss $ Contemporary Music Acquittal form 16d self-grantee

Profit/Loss $ Arts Grants Acquittal form 16c self-grantee

Profit/Loss $ Designer Fashion Acquittal form 16b self-grantee

No of programs No of performances, productins, exhibitions, publications, sales, tours, acitivties number


8 self-MPA

Other Standard of DCA service delivery

usefulness, accessibility, fairness, diversity and timeliness of DCA's service to grantees Likert scale Looking Forward Fund

Acquittal Form 17 self-grantee

DRAFT



ISSUES PAPER 2 - BENCHMARKING

AUGUST 2011

DRAFTDISCLAIMER



Document Control


v 1.0 Issues Paper 2 John Nell Michael Chappell 19 August 2011

23 Lyall Street, South PerthWestern Australia, 6151t (08) 9367 1511f (08) 9367 [email protected]

i

DRAFTCONTENTS

1 INTRODUCTION 1

2 KEY ISSUES AND CHALLENGES 2

2.1 A clear distinction between ‘impact’ and ‘value’ 2

2.2 Capturing Cultural Value and Quality 4

2.3 A ‘hierarchy’ of key outcomes? 4

2.5 Developing more fine-grained definitions of each category – reflecting DCA priorities 7

2.6 Measuring economic impacts through GVA based models 9

2.7 The fundamental principles of public value models 10

BIBLIOGRAPHY 11

ii

DCA - Issues Paper 2: Benchmarking

DRAFT

1

1.0DRAFT1 INTRODUCTION

This short paper is a benchmarking input into the early conceptual scoping of DCA’s new value framework.

In our project proposal we talked about the need to review international literature on best practice models and measurement to help aid the design of the new framework and help identify appropriate sources of peer review for the project.

This short paper shares some of that early review work, but with explicit reference to the emerging logic framework developed through early discussions between DCA and the Pracsys team. That framework begins to identify some of the key categories under ‘outcome’ and ‘impact’ areas that are of particular interest to DCA.

This short benchmarking analysis will comment on those dimensions of the logic model that are open to different interpretations and design modifications, using insights from international debate about the evaluation and measurement of public investment in the arts to help refine the emerging logic model.

In addition, this paper will seek to raise some broader issues arising from the benchmarking review that DCA needs to consider at this early scoping stage of the project – which impact both on the design of the value framework, but also on the downstream strategy for measurement and evaluation. In other words, it is important at this stage to not only benchmark the emerging framework, but to stress test the robustness of the model, and the conceptual clarity underpinning it.

With those aims in mind, this short document does not aim to be a comprehensive literature review – although key documents reviewed are listed in the bibliography. Rather the aim is to offer some indicative views, underpinned by key references, about the key issues and possible approaches that DCA need to work through with the Pracsys team over the coming weeks.

As we firm up the logic model, we can then benchmark in more detail some of the key elements and design options within the value framework.

Figure 1: Logic Framework Diagram

Source: Pracsys 2011; Productivity Commission, 2011

2

DRAFT2.0

Our benchmarking review has identified a number of higher order issues within the logic model that need to be addressed, as follows:

1. The need to build a clear distinction between outcomes / impacts – and value

2. To clarify current thinking about capturing cultural value, and quality, in the arts and cultural sector

3. To test whether DCA have in mind a clear ‘hierarchy’ of key outcomes, impacts and value measurements?

4. To begin to explore whether some of the categories in the logic model are actually subsets of other categories

5. To begin to develop more fine-grained definitions under each category – closely reflecting DCA priorities

6. To explore whether DCA wants to be more explicit in terms of capturing economic outcomes / impacts?

7. To restate the fundamental principles of public value models so that DCA can make some clear decisions about its overall public value approach

2.1 A CLEAR DISTINCTION BETWEEN ‘IMPACT’ AND ‘VALUE’

It may seem a strange place to start, but is alarming how often the distinction between ‘impact’ and ‘value’ is collapsed in debates about the returns on cultural investment. For example, as Bakhshi1 has recently observed:

1 Bakhshi, H. (2011) “Leadership and cultural value’ unpublished memeo

‘Economic impact is not the same as value. Economic impact refers to the measurement of the employment, output and productivity consequences of cultural activities. Properly executed, economic impact studies are essential for economic development agencies that see culture as a locus for, or as an instrument of, economic development.

Perversely, however, all the economic studies one sees in the cultural sphere tend to be of the economic impact variety. Next to none look at valuation, using the empirical tools endorsed by the Treasury’s Green Book – the government’s official guide to cost-benefit analysis – that public economists have deployed so successfully in other controversial areas like the environment and health where, like culture, a good deal of public value is not mediated through markets, and is therefore not captured through market prices and transactions.’

In other words, the ‘economic value’ of the arts, as opposed to the ‘economic impact’ of the arts, seeks to value the arts in the round, using the techniques and language of economics – for example through using stated preference techniques to explore what people value, their choices and their preferences, as reflected in a common unit of measurement (money).

So for example, if we take the example of so-called heritage assets (as per DCA’s KPI that ‘Western Australia’s natural, cultural and documentary collections are preserved, accessible and sustainable’), we would explore people’s willingness to pay to preserve the heritage in question. These non-use values may relate to the asset’s existence value (people value the existence of the heritage item even though they may not consume its services

2 KEY ISSUES AND CHALLENGES

3


DRAFTdirectly); its option value (people wish to preserve the option that they or others might consume the asset’s services at some future time); and its bequest value (people may wish to bequeath the asset to future generations). These non-use values are not observable in market transactions, since no market exists on which the rights to them can be exchanged.2

Willingness-to-pay studies of various sorts can be conducted at the micro-level to assess the community benefits from conservation of a specific heritage item. Alternatively they can be applied at the macro-level to find out how much the population at large would be willing to see spent out of its taxes on heritage protection in general.

If one looks at the UK experience, the Department of Culture, Media, and Sport (DCMS) approach has been to start with ‘impacts’ – and seek to understand the full range of potential impacts produced by the arts and cultural sector (in total the DCMS have produced value frameworks that contain around 90 different outcomes / impacts). Then DCMS, having decided which of these impacts are the most promising, has attempted to put a monetary value on those impacts. So for example, it has demonstrated the economic value of engagement in terms of the subjective well-being income compensation of the arts.3

2 Throsby, D. (2006) ‘The Value of Cultural Heritage: What can economics tell us? In ‘Capturing the Public Value of Heritage’ English Heritage.

3 See Selwood, S (2010.21) ‘Making a difference: the cultural impact of museums: An Essay for NMDC’ Sara Selwood Associates

Interestingly our review of international practice reveals that few, if any, Arts Councils have taken upon themselves the responsibility of consistently capturing these broader ‘value’ creating contribution of the arts (embracing not just economic value, but cultural, social, and public value as well). So a key question for DCA, given the cost of these evaluation techniques, is whether to make such considerations an integral part of the value framework. And then for DCA to be clear about the dynamics of that decision - namely to balance a desire to sharpen the rationale underpinning their investment decisions, and to make a stronger case for overall public investment in the arts within WA, against the resource cost of measurement and evaluation.

Experience from the UK suggests that contingent value / willingness to pay measures are hard to directly relate to allocation and investment decisions, which is why evaluations of this type are rarely commissioned.

These observations suggest two immediate questions for the DCA as they develop their value framework:

1. Which, if any, of the key outcomes/impacts (economic, social) would DCA like to prioritise and seek to capture measurements of their broader value?

2. What is the best approach to measuring these ‘value’ creating activities?

One approach would be to regularly commission stakeholder perceptions surveys, and public consultations (so for example, DCA could commit to carrying out a full public value assessment of its top twenty funded organisations every 5 years. Or a contingent value assessment of the same group of funded organisations on a similar timescale).

4


DRAFT2.2 CAPTURING CULTURAL

VALUE AND QUALITY

There are now an established range of methods to try and capture the qualitative cultural impacts of public investment, and in turn cultural value. These include self-evaluations; peer and user-review; and stakeholder analysis. There is also growing support for ‘artistic self-assessment.’ Bailey and Richardson4 have suggested that there is no ‘one size fits all’ approach to assessing what Australasians refer to as ‘artistic vibrancy’ (and what in the UK is referred to as ‘excellence’):

‘The idea that universal templates can be developed across companies and / or artforms is thus belied by empirical evidence, and the literatures shows, attempts to create such templates often degenerate into box-ticking’.

Methods suggested by their case studies, which include peer and staff opinion, could include assessment panels, guest artist surveys and staff days – reflecting the 360 degree review approach supported by much of the literature on performance management.5

In the UK, Arts Council England is currently rolling out an Artistic Assessment Scheme, that is centred on ‘artistic assessments by assessors who have knowledge and experience of the arts’, and focuses on ‘excellence’ rather than ‘impact.’

4 Bailey, J. & Richardson, L. ‘Meaningful Measurement: a literature review and Australian and British case studies of arts organisations’ conducting artistic self-assessment’ Cultural Trends, No. 76

5 See Selwood, S. (2010.22) op cit)

More generally, the issue of defining artistic ‘excellence’ remains a sticky issue once you step beyond the established approach of judging against appropriate standards as set by professional practice in the relevant artform, with those judgements taken by informed practitioners.6

A key issue for DCA is the level of commitment it wants to display in measuring quality, and the corresponding resource implications of adequately funding peer reviews of artistic vibrancy or excellence. For example, bringing in national and international peers is an important part of a robust process, due to the inherent danger of a ‘closed’ artistic community within a particular state or country indulging in ‘grade inflation’ to support each other’s claims for ongoing funding. The cost implications of such an approach are also self-evident.

2.3 A ‘HIERARCHY’ OF KEY OUTCOMES?

How far does DCA need to determine a hierarchy of core outcomes, and how is DCA expecting its funded portfolio to respond to those outcomes in their funding proposals.

Presumably DCA would not ascribe the same importance to all of the outcomes in the emerging logic model? So for example, audience reach is presumably more important than entrepreneurship across the sector? In other words is there a higher order goal set that needs to lie at the heart of the value framework – with other subsidiary, ‘nice to have’ impacts lying within the model, but as secondary

6 See McMaster, B. (2008) ‘Supporting excellence in the arts: from measurement to judgement.’ Department of Culture, Media and Sport; and Throsby, D. (2010) ‘The Economics of Cultural Policy’, Cambridge University Press

5


DRAFTsources of value creation compared to the higher order goal set.

The approach adopted by Arts Council England in their 10-year framework, ‘Achieving Great Art for Everyone’ is an interesting model for the DCA to benchmark against. Effectively ACE’s value framework is built around five 10-year goals – which are their prioritized outcomes (see Figure 2 below - I have included under each goal one example of ACE’s prioritized success criteria for the goals).

What is interesting about the ACE approach is that these five goals act as the specific impact areas which then directly form their investment criteria. So for example, regularly funded organisations in England (now called National Portfolio Organisations) had to show in the latest funding application round how they would make a significant contribution to at least two or more of the five goals.

So for DCA, this model raises some interesting questions. Firstly, how are DCA going to relate the value / investment framework to the application / evaluation process for funded organisations?

Secondly, are there any priorities in the ACE model that are not currently in the emerging DCA logic model, which could be usefully included? For example, there is currently no mention of a specific focus on children and young people, yet since March 2011 DCA has put in place a new ‘Action Plan for Young People and the Arts.’ Are children and young people a subset of the ‘community’ category under ‘Reach’?

Similarly, does the ‘entrepreneurship’ category under ‘Quality’ imply that DCA are looking for more sustainable business models – or does ‘entrepreneurship’ imply something else in terms of impacts. Similarly, do the ‘corporate’ and ‘philanthropy’ categories under ‘Reach’ imply something similar in terms of leveraged income / diversified business models.

Thirdly, the ACE model suggests something interesting about the different approaches DCA could adopt to the overall ‘architecture’ of the value framework. In addition to the five goals and corresponding priorities, ACE have a higher order set of outcomes they are seeking from their work and investments, as detailed in Figure 3.

Figure 2: ACE’s Ten Year Goals

Goal 1: Talent and artistic excellence are thriving and celebrated

• Using our investment to ensure excellent art happens

Goal 2: More people experience and are inspired by the arts

• Developing arts opportunities for people and places with the least engagement

Goal 3: The arts are sustainable, resilient and innovative

• Strengthening business models in the arts and helping arts organisations to diversify their income streams, including by encouraging private giving

Goal 4: The arts leadership and workforce are diverse and highly skilled

• Creating equal opportunities to enter the arts workforce

Goal 5: Every child and young person has the opportunity to experience the richness of the arts

• Improving the delivery or arts opportunities for children and young people

Source: ACE (2010) ‘Achieving Great Art for Everyone’

6


DRAFTFigure 3: ACE’s higher order outcomes

Excellence = outstanding art and outstanding art experiences

Reach = more people attending and taking part in the arts

Engagement = more people feeling the arts are meaningful to them

Diversity = the arts reflecting the diversity of contemporary England

Innovation = artists and organisations having the freedom, and being challenged, to innovate.

In some respects, having facilitated the ACE top team in producing ‘Achieving Great Art for Everyone’, it’s true to say that ACE ‘fudged’ the link between these higher order goals and the more granular ten-year goals / priority impacts – although they did check that the ‘outcome’ set produced by the five ten-year goals would deliver on these higher order goals.

In practical terms, these higher order goals (excellence, reach, engagement, diversity and innovation) are ACE’s touch stone ambitions in achieving their overall mission of ‘Great Art for Everyone’

We’re not suggesting DCA follow a similar formulation, but at the moment the emerging logic framework feels like it might be blending higher order outcomes (quality, reach), with more granular outcomes (philanthropy), and one of the things we need to work through is establishing exactly how DCA want that clear hierarchy of outcomes and related impact areas to look like, and how this relates to DCA’s current KPI set, or indeed a revamped KPI set.

International benchmarking suggests our collective aim should be to create a clear hierarchy of goals and performance management grids – i.e. conceptualizations that define the higher level goals of DCA that are then broken down into more specific and more concrete categories of activities and effects that constitute DCA’s particular strategy for creating public value through the arts. Our benchmarking work also suggests it is vital not to overcomplicate the goal hierarchies, with three to six key goals forming the core of any public agency’s understanding of itself and its mission.

In a fascinating study by Moore et al7 of 13 State Art Agencies in the USA, all of whom expressed in interest in developing a public value approach, the authors summarized the key higher-level goals of those state agencies – which is also a useful input for DCA to benchmark against (see Figure 4 below).

Figure 4: Key goals of US State Art Agencies in Moore study (2005)

• Contribute to state economic development

• Encourage and facilitate widespread participation in the arts

• Support and strengthen the arts community

• Strengthen the quality of individual and community life

• Help children learn and achieve

• Integrate arts activity in other state services

• Generate public policy support for the arts

• Generate popular support for public funding of the arts

7 Moore, M et al (2005) ‘Creating Public Value Through State Arts Agencies’ The Wallace Foundation

7


DRAFT2.4 CATEGORIES AND SUB-CATEGORIES

Following directly on from these observations about the ‘architecture’ of the goals, outcomes and impacts, we need to explore with DCA the extent to which some of the categories in the logic model are actually subsets of other categories, or other higher order goals.

So for example let’s look at one of DCA’s Key Performance Indicators – ‘A creative, sustainable, and accessible culture and arts sector.’ If we take the financial health measure of ‘sustainability’ within DCA’s KPIs, and then look at the emerging logic framework – the focus on corporate leverage, philanthropic leverage, and entrepreneurship, are potentially all sub-sets of delivering on the financial health element of the sustainability KPI.

This could have a number of implications as we refine the logic framework. Firstly, should DCA have a higher order outcome goal around the ‘sustainability’ of the funded arts sector in WA? Secondly, irrespective of the higher order goal issue, all of this suggests that DCA might want to revisit the current definition of financial health used within their KPIs (reserves greater than 20 per cent of turnover; working capital greater than 2.0; and case reserves of greater than 10 weeks of expenditure).

For example DCA could look at leverage ratios, getting funded organisations to detail all their funding streams in order to calculate how they are leveraging public funds to generate other revenue in terms of corporate and private income. Or alternatively, DCA could decide to look at the balance of income streams for each of its funded organization.

In this context within the UK there has been a lively debate across the sector about the ‘gold standard’ mixed economy ratio for publicly funded arts organisations suggested by Arts and Business – comprising of a third public subsidy, a third earned income, and a third through sponsorship and philanthropic giving.8 This may be an area in which DCA could choose to innovate in terms of the new value framework.

2.5 DEVELOPING MORE FINE-GRAINED DEFINITIONS OF EACH CATEGORY – REFLECTING DCA PRIORITIES

DCA and the project team face quite a large number of prioritisation choices under each of the emerging categories. For example, if we take the current ‘social’ category under ‘reach’ – this ‘social’ category could include a wide range of outcomes / impacts – from educational and health based outcomes, through to strong and cohesive communities. Depending on how many outcomes DCA wish to prioritise, DCA could choose to direct us to review the best practice literature about how best to measure the wider impacts of its investments, including Social Return on Investment (SROI) approaches.9 For example, within the new value framework, DCA could choose to invite applicants to give greater consideration to forecasting their SROI if they meet their intended outcomes as a result of a successful funding application to DCA’s new value framework.

8 Arts and Business (2010) ‘A Private Sector Policy for the Arts’9 Cabinet Office (2009) ‘Social Return on Investment – an

introduction’

8


DRAFTSimilarly, the ‘engagement’ category under ‘Reach’ could be a narrow measure of access and audience numbers, or a more innovative measure of deeper public participation and consultation, aligned with DCA’s Strategic Objective 2 (‘DCA objectives are aligned within a pubic value framework’)10 and building on Australia Council’s ‘Creative Communities’ initiative.

This raises the key issue as to whether DCA already has in mind a hierarchy of priorities within each sub category – and is DCA clear about how is it arriving at those prioritized sub-categories? If one looks at international approaches, the route to such prioritized outcomes tends to reflect two key influences. Firstly, a pragmatic focus on where the direct causal link from cultural investment to these outcomes is strongest (for example see the CASE evidence programme in the UK and the strong link between cultural investment and measures of subjective well-being).

Secondly, because those outcomes are priorities of either Central of Local Government, and as a result are actively shaping the broader commissioning of public services, hence opening up additional public budgets that cultural organisations could tap in. No doubt DCA will be balancing these strategic and tactical considerations.

So for example, in the UK the ‘Creative Partnerships’11 experience for cultural organisations, and the ‘Renaissance Programme’ led by the Museums and Library Association (MLA)12, has led directly to better links between schools and cultural organisations, and a

10 Department of Culture and the Arts, ‘Strategic Plan 2010-2014’11 See www.creative-partnerships.com12 See www.mla.gov.uk/what/programmes/renaissance

growing emphasis that cultural organisations should be doing more to formalize their education offer to schools, and in turn that cultural funders and Schools themselves need to be clearer about the outcomes they are expecting from commissioning services from cultural organisations. Clearly, given DCA’s ‘Creative Connections Partnership Framework’ with the Department of Education, this might be one of the outcome areas DCA may choose to prioritise

More broadly, a key factor will be factoring in some inevitable path dependency in the new value framework as a result of DCA’s existing KPI set. For example, the ‘creative’ outcomes element of the current KPI set is an obvious area where the new value framework will suggest some innovations. Which begs an inevitable question. DCA’s current strategic plan ends in 2014. As a result of developing this new public value framework, will there be an opportunity to renegotiate the Department’s KPI’s to align them more closely with the new value framework?

As a development team, we need as much clarity as possible around what’s up for grabs and what isn’t within the new framework.

9


DRAFT2.6 MEASURING ECONOMIC

IMPACTS THROUGH GVA BASED MODELS

Our benchmarking review suggests that DCA will need to develop more clarity on its approach to capturing the economic impact of its investments in the sector.

At the moment the emerging logic model doesn’t make any specific provision for GVA type measures (although they are no doubt implied within the direct /indirect impact categories relating to communities). This raises the question as to whether economic impacts – measured in GVA terms – need to more clearly sketched and differentiated. In other words, DCA need to consider whether such measures are of primary or secondary importance to the value framework.

Our review of the international literature confirms there is a maturing approach to measuring economic impacts, with an increasing emphasis on capturing Gross Value Added (GVA) impacts.13 In simple terms GVA models attempt to measure within a defined geographical area the total value of goods and services available through economic activity. So for example, if additional jobs are created in a regional economy, this will lead to higher GVA. Similarly higher salaries and business profits will lead to a higher GVA figure.

13 There needs to be a clear distinction between the contribution of the arts to the economy (measured in GVA terms), and the ‘economic value’ of the arts, which seeks to value the arts in the round, using the language of economics – for example through using stated preference techniques to explore what people value, their choices and their preferences, as reflected in a common unit of measurement (money)

All GVA models are therefore essentially variations on input/output models that are trying to capture:

• Direct and indirect employment effects (for example net employment gains; wages)

• Profits (gross operating surplus)

• Secondary / external visitor effects

• Indirect / Supply Chain effective (income multiplier effects through local expenditure in goods and services)14

In deciding its approach to capturing economic impacts, DCA will need to determine its preferred approach to capturing such outcomes and whether to adopt a formal input / output model to measure overall GVA impact (see Figure 5 for example).

There is also the issue here of the balance between ‘tight’ and ‘broad’ definitions / approaches. If we depict GVA measures as a ‘tight’ definitional approach, there has understandably been a growing interesting in ‘broader’ approaches – charting the contribution of the arts to that wider creative economy (including place making and brand considerations, which often find a quantitative expression in City benchmarking exercises such as the Arnholt-GfK Roper City Brand Index15, which measures issues such as ‘pulse’, and ‘liveability’ in cities, which include consideration of the cultural offer and creative milieu).

14 See Office for National Statistics (2010) ‘Measuring the economic impact of an intervention or investment’

15 See www.simonanholt.com/Research/cities-index.aspx

10


DRAFTFigure 5: The Shellard Model for measuring economic impact

The Shellard Model (below) grew out of Arts Council England commissioned research on the economic impact of performing arts organisations. The challenge for AGMA is to generate a model that is serviceable to a wide range of potential organisations across the portfolio.

Economic impact = annual turnover + overseas earnings + additional visitor spend + salaries + subsistence allowances + goods and services expenditure X a multiplier of 1.5

The multipliers applied to such models differ – for example the Treasury Green Book uses a regional multiplier of 1.0 for affects in the regional economy*

* See Department for Business, Innovation and Skills (2009) ‘Practical Guidance on Implementing the Impact Evaluation Framework’

2.7 THE FUNDAMENTAL PRINCIPLES OF PUBLIC VALUE MODELS

The ‘public value’ literature is well known and we won’t rehearse the full logic model here.16 But given there is a lively debate in the literature about what ‘public value’ is, this suggests that DCA have some clear strategic choices about which aspects of public value they want to emphasise.

As Gray notes17, the literatures implies that public value is at least one of:

• An approach to management

• An end-product of the management process

16 See Moore (1995); Moore et al (2005); Denhardt & Denhardt (2000); Coles & Parston (2006); and Blaug et al (2006)

17 Cray, C. ‘Arts Council England and Public Value: A Critical Review’ De Montford University

• A set of processes that organisations could/should/ought to pursue

At the heart of public value approaches is the desire to both improve service delivery (thus delivering improved ‘value’ to the public), and to develop a better linkage between public sector organisations and the general public so that there is a better match of expectations between the two.

It is clearly important that DCA are clear about the key benefits they want to deliver through a public value approach. These might include:

• Enhanced public legitimacy for its actions and decisions (in which the WA public will better understand your activities and decisions, and / or exercise more influence over your activities and decisions

• Improved service delivery, which may or may not include greater public involvement in this delivery through the identification of service needs18

As this brief review makes clear public value approaches can therefore encompass provision, outcomes, trust or cost-effectiveness? Are some elements of these public value models more important to DCA than others, and how will this shape DCA’s measurement and evaluation of public value outcomes?

18 So for example, the Kentucky Arts Council include within its strategy goals the ambition that ‘Public Policy is favourable to the arts in Kentucky’, and that they will ‘encourage citizens to become involved in shaping public policy (see Moore (2005))

11

1.0DRAFT

Arts and Business (2010) ‘A Private Sector Policy for the Arts’

Arts Council England (2010) ‘Achieving Great Art for Everyone’

Arts Council England (2011) ‘Understanding the economic contribution of the arts: Research Project Definition Document’ Unpublished draft

Bailey, J. & Richardson, L. ‘Meaningful Measurement: a literature review and Australian and British case studies of arts organisations’ conducting artistic self-assessment’ Cultural Trends, No. 76

Bakhshi, H. (2010) ‘Beauty: Value Beyond Measure’ CABE

Bakhshi, H. et al (2009) ‘Measuring Intrinsic Value: How to stop worrying and love economics’ Mission Models Money

Bakhshi, H. (2011) “Leadership and cultural value’ unpublished memeo

Blaug, R. et al (2006) ‘Public Value, Politics and Public Management: A Literature Review’ The Work Foundation.

Bunting, C. (2010) ‘Culture and Sports Evidence Programme: The impacts of engagement: a systematic review of the research on learning outcomes for young people participating in the arts.’ Arts Council England

Bunting, C, et al (2010) ‘Achieving Great Art for Everyone: A review of research and literature to inform the Arts Council’s ten year strategic framework.’ Arts Council England

Department for Business, Innovation and Skills (2009) ‘Practical Guidance on Implementing the Impact Evaluation Framework’

CASE (2010) ‘Understanding the value of engagement in culture and sport’ Department of Culture, Media and Sport.

Cabinet Office (2009) ‘Social Return on Investment – an introduction’

BOP (2011) Edinburgh Festivals Impact Study. Edinburgh Festivals

Coles, M. & Parston, G. (2006), ‘Unlocking Public Value’ (London, John Wiley and Sons)

Denhardt, R. & Denhardt, J. (2000) ‘The New Public Services: Serving Rather Than Steering’, Public Administration, Vol 60, pp. 549-59

Garcia, Beatriz; Melville, Ruth and Cox, Tamsin (2010) ‘Creating an impact: Liverpool’s experience as European Capital of Culture’. Impacts 08

Jura (2005) ‘Bolton’s Museums, Libraries and Archives Services - An Economic Valuation’. Bolton Metropolitan Borough Council, MLA North West

Moore, M (1995) ‘Creating Public Value: Strategic Management in Government’ Cambridge, Harvard University Press

Moore, M et al (2005) ‘Creating Public Value Through State Arts Agencies’ The Wallace Foundation

Morris Hargreaves McIntyre (2009) ‘Cultural Impact Study - The impact of the arts in Birmingham’. Birmingham Arts Partnership

BIBLIOGRAPHY

12


DRAFTOffice for National Statistics (2010) ‘Measuring the economic impact of an intervention or investment’

Reeves, Michelle (2002) ‘Measuring the economic and social impact of the arts: a review.’ Arts Council England

Roger Tym & Partners (2008) ‘Assessment of the contribution of museums, libraries and archives to the visitor economy.’ MLA

S4W (2010) ‘Arguments for a stronger role for the Arts in Economic Development and Regeneration’. Arts Council England

Selward, S. (2010) ‘Making a difference: the cultural impact of museums: An Essay for NMDC’ Sara Selwood Associates

Shellard, Dominic (2004) ‘Economic impact study of UK theatre’. Arts Council England

Throsby, D. (2006) ‘The Value of Cultural Heritage: What can economics tell us? in ‘Capturing the Public Value of Heritage’ English Heritage.

University of Bournemouth (2011) ‘Destination: Music – The contribution of music festivals and major concerts to tourism in the UK’. UK Music

DEPARTMENT OF CULTURE AND THE ARTS FUNDING … · o#ers a complete map of the process of public...

Documents

Transcript of DEPARTMENT OF CULTURE AND THE ARTS FUNDING … · o#ers a complete map of the process of public...