Mass Engaging Report

8/4/2019 Mass Engaging Report

1/119

3masslbp.com

Engaging with impact:Targets and indicatorsfor successfulcommunityengagement by

Ontarios Local HealthIntegration Networks

A citizens report

from Kingston,Richmond Hilland Thunder Bay

r.1


2/119

Engaging with impact:Targets and indicatorsfor successful

communityengagement byOntarios Local HealthIntegration Networks

A citizens reportfrom Kingston,

Richmond Hilland Thunder Bay

We know that community engagement

matters especially to our public

health system. As Ontarios Local Health

Integration Networks strengthen their

focus on community engagement,

what are the common benchmarks

and commitments that citizens thinkmatter most?

Engaging with Impact addresses the

challenge of evaluating engagement

and proposes a series of indicators that

can be used to assess performance and

develop a culture of engagement across

Ontarios public health system.

This report features:

A special essay on the challenge of

evaluating deliberative engagement by

Professor John Gastil;

Two essays comparing the commitment

of Canadian and UK health systems to

greater community engagement by the

Wellesley Institute and the British think

tank, Involve;

Interviews with the directors responsible

for community engagement in the North

West, Central and South East LHINs;

An account of three Citizens Workshops

that provide the basis for the

recommendations in this report;

An engagement scorecard for OntariosLHINs which proposes principles,

recommendations and indicators.

MASS LBP is reinventing

public consultation

masslbp.com

This project was commissioned by the Ontario Ministry of

Health and Long Term Care, Health System Strategy Division

and the Central, North West and South East LHINs.

r.1 Local Health IntegrationNetwork


3/119


4/119


5/119

MASS LBP is a new kind o company

that works with visionary governments

and corporations to deepen and improve

public consultation and engagement.

We design impartial and ully transpar-ent public learning processes that build

awareness, consensus and insight.

MASS LBP provides an unparalleled

range o consultation and engagement

services or government, corporate and

not-or-prot clients. From conception

to execution to evaluation, MASS LBP

delivers highly innovative engagement

strategies that increase public under-standing, legitimacy and support.

Our services include:

comprehensive process design and

delivery rom 20 to 200+ participants

strategic advice, analysis and recom-

mendations concerning eective

public engagement and stakeholder

consultationcorporate and public needs

assessment

program evaluation and analysis

custom research and dissemination

acilitation and learning

public communications and curriculum

development

event coordination and logistics

We regularly make presentations to

audiences about our work concerning

the uture o responsible government,

public systems design and civic engage-

ment. We also oer seminars and relatedprogramming to clients on many o these

themes.

Inspired by Canadas rst Citizens

Assemblies, MASS LBP was ounded

in 2007 by Peter MacLeod and George

Gosbee to extend this model and

reinvent public consultation.

MASS LBP is based in Toronto, withassociates in Vancouver, Ottawa and

London, U.K.


6/119

First published in 2009

MASS LBP.

Some rights reserved.

392A King Street East

Toronto, OntarioCanada M5A 1K9

1 800 369 7136

masslbp.com

ISBN 978-0-9811005-0-1

Series design by Concrete Design

Set in Helvetica Neue

and Berthold Baskerville Book

Cover paper: Cascades Cover 80

Text paper: Cascades Natural 60(FSC Mixed Sources)

Open access. Some rights reserved.

As the publisher o this work, MASS LBP

wants to encourage the circulation o our

work as widely as possible while retain-ing the copyright. We thereore have an

open access policy, which enables any-

one to access our content online without

charge. Anyone can download, save,

perorm or distribute this work in any or-

mat, including translation, without written

permission provided that the ollowing

conditions are met:

MASS LBP and the author(s) arecredited.

This summary, the MASS LBP com-

pany prole and the address

www.masslbp.com are displayed.

The text is not altered and is used

in ull.

The work is not resold.

A copy o the work or link to its use

online is sent to MASS LBP.

MASS LBP grateully acknowledges the

work o Creative Commons in inspiring

our approach to copyright. To nd out

more, go to:

www.creativecommons.org


7/119

4Chapter Title


8/119

5masslbp.com 5masslbp.com

Contents

Section 1: Evaluating Engagement

Introduction

Section 2: Understanding Engagement

A Comprehensive Approach to Evaluating Deliberative Public Engagement

The Canadian Experience: Observations and Lessons from the Canadian

Health Sector

The English Experience: Evaluating Patient and Public Engagement in Health

Learning from the LHINs

Working with the Citizens Workshops on Health and Engagement

Section 3: A Scorecard or Evaluating Engagement

A Scorecard for Evaluating Engagement

Appendices

Denitions of Community Engagement

Members of the Citizens Workshops on Engagement and Health

Indicators and Ideas Exercise

Endnotes

7

15

28

43

59

71

93

100

102

103

110


9/119

Section 1:EvaluatingEngagement


10/119

7masslbp.com

Engaging with impact: Targets and indicators or successul community

engagement by Ontarios LHINs ocuses on the value o community

engagement. Specically, it deals with the challenge o evaluating

engagement and proposes a series o recommendations and indica-

tors that can be used to assess perormance and develop a culture o

engagement that will help to rewrite the relationship between health

administrators and their public.

Local Health Integration Networks were created in 2006 with

an explicit mandate to engage stakeholders and their communities.

More than this, the idea o engagement was central to their rationale.

Proponents o the LHIN system argued that regional planning authori-ties would be better positioned than ministry ocials to assess and

interpret local needs. LHINs could do this because they would be in

closer contact with the communities they served and because o the

strength and number o local relationships they could orge and sus-

tain.

Many o Ontarios LHINs have spent their rst three years dem-

onstrating the easibility and merit o this rationale. Using their ownexpertise and intuition and sometimes relying on simple trial and error,

they are working to better engage stakeholders and members o the

public and to connect their eorts to other planning and integration

processes.

For these organizations, the debate concerning the value o

engagement has largely been settled. In its place is the growing

recognition that a commitment to integrating engagement into the

abric o their organization requires upending many o the traditionalassumptions that have dened health systems planning and public

administration.

The capacity to engage with Ontario communities is one o the

Introduction


11/119

8Evaluating Engagement

LHINs dening and most distinctive competencies. As this com-

petency evolves, it promises to change how LHINs respond to the

interests and needs o their communities and to gradually transorm

how health systems planning is perormed. In this way, Engaging with

Impact is addressed to those who believe that community engagement

can be a major driver o health systems reorm.

Despite its length, this report is not exhaustive. Instead, we hope

it is stimulating and useul a rst installment amidst a broad range o

research and initiatives that the Ministry o Health and Long-Term Care

and the LHINs have begun to seed. Our purpose is to oer recommen-

dations and to be ameliorative rather than denitive a purpose that

is consistent with the sentiments o our citizen-participants, who in the

course o their work clearly understood that something as complexand amorphous as creating better systems o engagement rarely sub-

mits to single measures or immediate solutions.

Engaging with Impact begins with an essay by Proessor John

Gastil rom the University o Washington, one o North Americas

leading theorists concerned with the value o community engage-

ment. Gastil tackles the particular challenge o evaluating deliberative

engagement, where citizens and experts work together to examine and

solve problems. The essay oers ruitul reading or anyone wrestling

with the heightened challenges associated with designing, managing

and evaluating intensive engagement processes.

Subsequent papers rom the British think-tank Involve and

Canadas Wellesley Institute provide a comparative perspective on

evaluating engagement within centralized and decentralized

health systems.

Involve looks at the eorts being made by the English NationalHealth Service (NHS) to promote and evaluate engagement. In

England, a system-wide standard requires health service providers to

engage with patients and the public at large. Their paper examines the

ecacy o this standard and describes three initiatives that exempliy

the NHSs attempts to measure the outcomes o their investments in

public engagement.

The Wellesley Institute surveys a range o health agencies in

Canada and provides our examples o localized innovation. Without

a national standard or champion, interest in incorporating community

engagement has only recently begun to mature in Canada. Their paper

describes the challenges that need to be overcome or community


12/119

9masslbp.com

engagement to become more deeply embedded within Canadian

health agencies.

A ourth paper looks at the experiences o our three sponsoring

LHINs, each located in a dierent part o the province and conronting

very dierent demographic pressures. Interviews with the directors o

planning, integration and community engagement at the North West,

Central and South East LHINs describe their on-the-ground eorts to

build a local practice o engagement that inorms the operations o

their organizations and the decisions o their boards.

The second section o this report describes three Citizens

Workshops that were hosted in Kingston, Richmond Hill and Thunder

Bay between November 27 and December 6, 2008. These workshops

involved representatives rom each o the sponsoring LHINs, as wellas independent experts who oered their insight on the strengths and

weaknesses o the health system. Billed as an opportunity to learn and

contribute ideas or improving engagement, more than 3,000 invita-

tions were mailed to randomly selected households in each region.

Ultimately, 80 citizens came orward to participate during the daylong

events. Their work culminated in a series o presentations that are the

basis or the principles that underlay the evaluation scorecard ound at

the end o this report.

Creating a culture o engagement

The overarching theme o this report is how to create a culture o

engagement. It is a culture that LHINs, unique among the wider health

sector, are singularly able to develop and it is a culture that citizens

want and increasingly expect.

During the workshops, the inseparable nature o engagement and

integration also became clear. At its core, integration requires a will-

ingness to try new things, in new combinations. While it is easy to get

lost in the technical minutiae o integration agreements, it is harder to

remember that integration is rst and oremost an act o imagination

and guiding that imagination should be a common sense o purpose

a desire to improve the quality and eciency o health services

available to Ontarians. LHINs need to engage the imagination o the

public and their health service providers i they are to achieve theirobjectives or health reorm.

In order to create a culture o engagement that helps the LHINs

move towards these goals, we urge the ollowing:


13/119


1. LHINs as health service providers: Community Engagement generates

real health outcomes.

Community engagement is not peripheral but central to the work o

Ontarios LHINs. It is the service they provide to the health system ingeneral and to citizens in particular. LHINs must continue to invest

in their ability to provide and enhance this service. LHINs should be

at the centre o an ongoing and lively conversation about the values,

views and priorities o their stakeholders and the public at large.

Building such relationships will help to rebuild citizens trust in the

health system, nd new opportunities or integration and increase the

sense o shared ownership and responsibility or the perormance o

the health system as well as or the populations general health and

well-being. In this sense, the LHINs are health service providers, and

the service they provide is community engagement.

2. Citizens are ready and waiting.

Citizens are willing, capable and ready to make important contributions

to the work o Ontarios LHINs. However, the opportunities or citizens

to make a contribution either directly or indirectly remains limited and

episodic. The proessional expertise o health service providers and

the input and interest o the public are integral assets that cannotaord to be let on the table. LHINs need to work to make engagement

a routine and more visible part o their repertoire. Moreover, because

good communication is a precursor to eective engagement, LHINs

should work to align their communications and engagement strategies.

3. To harness public input, emphasize learning.

Most citizens are unamiliar with the inner workings o the health care

system and, consequently, with the work o Ontarios LHINs. But citi-zens are not only willing and ready to make a contribution, they are

also eager to learn. They want to become better inormed and they

want to better understand a system they rely on and value. LHINs can

add value to public input by creating opportunities or the public to

become better inormed. With this in mind, LHINs need to ask or more

than public opinion they must help citizens understand the nature

and constraints o their health care system or any other issue they are

being asked to address.

4. Make it real.

Facing many competing pressures and demands, citizens have a

good sense or the value o their time. They will engage most deeply


14/119

11masslbp.com

and meaningully when something is real and at stake. Their commit-

ment will always be proportionate to their sense o infuence and the

likely impact o their contribution. In this way, LHINs will only get out

o their eorts at engagement what they are prepared to put in. As

our scorecard explains, this means clearly dening the purpose o an

engagement and the role the public is expected to play. It means being

accountable and responsive to the publics contributions.

5. Focus on creating t-to-purpose engagement.

LHINs need to expand their repertoire and work to create a better t

between the processes they use to engage the public and the out-

comes they expect. In this report, we propose three classications

that describe the characteristics and the objectives o a wide range o

engagement processes.

6. Community engagement is mission critical.

Successul engagement is a key to meeting the LHINs objectives or

health systems reorm and unlocking the trust, imagination and com-

mitment o health service providers and the public. Poorly designed,

incomplete or insincere eorts to engage will only uel cynicism and

estrangement. Learning how to engage with impact is essential or

system-wide transormation.

Conclusions

The act that the LHINs have a clear mandate to invest in community

engagement demonstrates that the health system is eager to respond

to the concerns, needs and desires o citizens. Translating this man-

date into an eective culture o engagement should be a major ocus

o the LHINs over the next three years.

To help achieve this culture o engagement, the ministry should:

requirededicatedprogrambudgetsforthepurposeofengagingcommunities on substantive and ongoing issues

recognizeandrewardinnovationinengagement evaluatetheprogressofeachLHINtoimproveitseffortstowards

this goal

The LHINs should:

createengagementplansthatsupportandarecongruentwiththeir strategic objectives

diversifyanddeepentheirrangeofengagementofferings aligntheircommunicationsandengagementstrategies


15/119


encourageHealthServiceProviders(HSP)todeveloptheirownengagement plans and integrate these plans with their core oper-ations

Just as preventative health is about ordinary citizens taking control o

their lives, engagement is about the capacity o citizens to contribute

to the systems that serve them. I there has been a change o philoso-

phy rom reactionary to proactive health care provision, an analogous

philosophical shit is required to revolutionize the way health care

systems work and respond to the needs o citizens. In this light, com-

munity engagement is not just a task to be completed. It is an ongoing

process through which health outcomes are improved, trust is built,

public legitimacy is enhanced and systems transormation can

be pursued.


16/119

13masslbp.com


17/119

Section 2:UnderstandingEngagement


18/119

15masslbp.com

I you turned back the clock just 20 years, it would be dicult tond a person in public oce, academia or civil society talking about

the virtues o citizen deliberation. At that time, a ew innovativepublic deliberation programs, such as the Citizens Jury in the UnitedStates and the Planning Cell in Germany, existed, but they did soin an unortunate kind o isolation, sometimes overlooked even bythose who would develop deliberative programs o their own in thecoming years.1

Today, the landscape could not be more dierent. Growinginterest in citizen engagement has spurred a prolieration o new,

more sophisticated deliberative practices designed to ellicit substan-tive public involvement in policy-making and public aairs. Now theissue or planners and administrators isnt scarcity but choice.

With dierent agencies and organizations deploying diverseapproaches to deliberative citizen engagement, it has become moreimportant than ever to take seriously the evaluation o these var-ied processes. It is not pessimistic to say that we currently have nosystematic comparisons o alternative deliberative methods, thoughmany civic reormers, researchers and agency ocials have ideasabout when to use one process instead o another. To improveour knowledge o deliberation and upgrade the practice o citizeninvolvement, we must begin to evaluate the design, process and out-comes o our civic engagement activities.2

In this chapter, I aim to provide the tools necessary or doing so.I begin by clariying the meaning o deliberative public engagement

and discussing broad evaluation categories. I review each evaluationcriterion and suggest measurement tools and then conclude with asummary recommendation or conducting evaluations.

Proessor John Gastil

Department o Communication, University o Washington

A Comprehensive Approach to EvaluatingDeliberative Public Engagement


19/119

16Evaluating Deliberative Public Engagement

Dening Deliberative Public Engagement

It is imperative that reerences to deliberative public engagementconvey a suciently specic meaning so we can distinguish it romgeneric public involvement processes, such as ormal hearings orinormal consultations. For the purpose o this chapter, I dene this

term as an ocial or quasi-ocial process whereby policy-makers,policy/scientic experts and lay citizens work together on a publicproblem or concern, with the citizens careully examining a problemand seeking a well-reasoned solution through a period o inormed,inclusive and respectul consideration o diverse points o view.3

Breaking this down, the players in a deliberative public engage-ment need to include (1) appointed or elected ocials with some

degree o authority, (2) persons with content-relevant expertise and(3) lay citizens, whether randomly selected or otherwise recruited ina ashion that seeks diverse members o the general public. The citi-zens are at the heart o the process, but public ocials typically serveas the catalyst or initiating the deliberation and acilitating the imple-mentation o its ndings. The experts play a role behind the scenes(e.g., preparing brieng materials) or as personal resources thatcitizens can call on in the course o their deliberations (e.g., as key

witnesses). Together, the interplay o these participants constitutes apublic engagement process.

For such a process to be deliberative, it must meet a higher stan-dard or the quality o the dialogue, debate, discussion and other talkin which citizens participate. Table 1 shows a denition o a delibera-tive public meeting that I have ound helpul. First, a deliberativemeeting involves a rigorous analytic process, with a solid inorma-

tion base, explicit prioritization o key values, an identication oalternative solutions (sometimes pre-congured but oten still subjectto amendment) and careul weighing o the pros and cons. (Researchon group decision-making has ound that o these analytic elements,careul consideration o cons is oten the key to a high-quality pro-cess, and the emphasis on hard choices and trade-os in manydeliberation processes refects this.)4

Exclusive ocus on problem-solution analysis, per se, would

make our conception o deliberation overly rationalistic and over-look the social aspect o deliberation. One might say that the socialcomponent o deliberation is what makes it democratic deliberation,by requiring equal opportunity, mutual comprehension and consid-


20/119

17masslbp.com

eration, and respect. The social requirements also make clear theimplicit emphasis on inclusion and diversity in deliberation.5

Analytic Process

Social Process

Combine expertise and professional research with

personal experiences to better understand the

problems nature and its impact on peoples lives.

Prioritize the key

values at stake.

Create a solid

information base.

Integrate the publics articulation of its core values

with technical and legal expressions and social,

economic and environmental costs and benefits.

Identify a broadrange of solutions.

Identify both conventional and innovative solutions,including governmental and non-governmental

means of addressing the problem.

Weigh the pros, cons

and trade-offs

among solutions.

Systematically apply the publics priorities to the

alternative solutions, emphasizing the most

significant trade-offs among alternatives.

Make the bestdecision possible.

Identify the solution that best addresses the problem,potentially drawing on multiple approaches when

they are mutually reinforcing.

Adequately

distribute speaking

opportunities.

Mix unstructured, informal discussion in smaller

groups with more structured discussion in larger

groups. Create special opportunities for the reticent.

Ensure mutual

comprehension.

Ensure that public participants can articulate general

technical points and ensure that experts and officials

are hearing the publics voice.

Consider other

ideas and

experiences.

Listen with equal care to both officials and the

general public. Encourage the public to speak in their

authentic, unfiltered voices.

Respect other

participants.

Presume that the general public is qualified to be

present by virtue of their citizenship. Presume

officials will act in the publics best interest.


21/119


General Evaluation Criteria

The question o whether a process even aspires to approximatedeliberative citizen engagement precedes any serious attempt atevaluation. Ater all, one can justiy the considerable eort evalua-tion requires only i the process being examined claims to be (or has

some reasonable expectation o being) related to the particular idealso public engagement and deliberation. Presuming that the delibera-tive engagement program, project or event aspires to these ideals,then the ollowing our evaluative criteria are appropriate or assess-ing its overall quality on these terms.6

When implemented, a deliberative public engagement processshould be evaluated on its own terms. That is, the best way to judgeits eectiveness is to assess the extent to which it achieves the goalsthat such a process strives to achieve. Because deliberative engage-ment programs share some common ideals, however, they do sharea concern with (1) design integrity and (2) sound deliberation andjudgments. Ater all, such programs ail immediately i their designor the ensuing deliberation does not meet basic requirements, asdescribed below.

In addition, these engagement processes can be assessed in terms

o the outcomes their public events engender. Here, more varia-tion occurs among dierent programs, and the third criterion thusrequires (3) infuential conclusions and/or actions. For some pro-cesses, it will be enough or deliberation to yield recommendationsthat carry infuence, whereas other programs will emphasize takingdirect action, whereby citizens not only talk but work together toexert their infuence.7

Finally, the greatest variation in purposes comes rom the

wide range o (4) additional benets or public lie that delibera-tive engagement processes hope to realize. Herein, I will considermethods or evaluating a range o these, rom benecial eects onindividual citizen participants to broader impacts on the communityor even the larger political culture. I call this nal criterion second-ary benets because it reaches beyond the immediate purpose andimpact o citizen deliberation, but nearly every deliberative enterprisecarries ambitions that extend outward in this way.8

The sections that ollow consider these our criteria in greaterdetail and suggest the eective means whereby one might assess theaccomplishment o each.


22/119

19masslbp.com

Criterion 1: Design Integrity

A high-quality deliberative engagement process gains its power partlyrom the integrity o its development, design and implementation.This criterion can be broken down into three more specic sub-components:

1. Unbiased raming. The process by which issues are ramed ordeliberation should be transparent, subject to open criticism byall interested parties. The resulting issue rame should be a airrepresentation o conficting views and arguments. Even when theorganizers imagine that they have an undened, open issue rame(e.g., political reorm, without speciying any options), its stillthe case that they selected that issue and generated language todescribe it.

2. Process quality. The deliberative procedures themselves shouldbe developed in consultation with (or at least subjected to commentrom) interested parties, particularly those with dierent points oview on the issue at hand, and the resulting process should beconsistent with the best practices or deliberation (e.g., rigorous ana-

lytic process or studying the problem and generating and evaluatingsolutions, along with respectul and egalitarian relationsamong participants).

3. Representative. The selection o citizen participants should givebroad opportunity to all potentially interested parties (excluding onlythose with public oces or unusually high personal/nancial stakesin an issue). The resulting body o citizen participants (hereater

called a citizen panel) should prove representative o the generalpopulation and, in particular, include representatives rom any per-manent minorities (i.e., groups or whom public policy consistentlygoes against their interests) and even smaller-numbered culturallyrelevant identity groups (i.e., sub-publics or communities who seekvisible representation in any public deliberative body).

One can assess these design eatures through direct inspectiono relevant event and design records, along with interviews withorganizers and interested third parties. Specically, I recommend theollowing evaluation methods:

1. Evaluating the issue rame. Whenever possible, the issue rames


23/119


airness should be evaluated beore the deliberative body convenesand reaches its conclusion. This way, evaluations will not refectreactions to the outcomes. A neutral third party (e.g., unaliateduniversity researcher or program evaluation specialist) can evaluateindependently, through inspection o project documents and proce-

dures, whether the raming process was neutral and transparent, butideally this process is evaluated by interested parties rom all relevantperspectives. The latter approach oers a more varied perspective onthe procedures airness to the particular concerns o dierentinterest/advocacy groups.

2. Evaluating process quality. This ollows the same basic protocol asissue rame evaluation, with two exceptions. It is useul to get prelim-inary process assessments beore deliberation begins, but wheneverpossible it is helpul to complement these with assessments duringand ater deliberation. The actual implementation o the delibera-tive procedures may shape the nal evaluations thereo. To ensurecommensurate evaluations, it is also important to discuss with eachevaluator including interested parties the conception o delib-eration underlying the process design. (This parallels the present

chapters eort to careully dene deliberation.)

3. Assessing representativeness. The nal body o citizens whoattend the event (versus those who register or pledge to attend)should be surveyed to determine their relevant demographic andideographic (attitudinal) characteristics. These characteristics canthen be compared against relevant census and survey data or thetargeted geographic/political region. This can be more expensive

when the target area does not have a readily available census orsurvey prole, as in the case o a watershed, transit area, biozone orother non-standard region.

Criterion 2: Sound Deliberation and Judgment

Beyond their process eatures, deliberative civic engagement pro-grams should show signs o high-quality judgment. Thus, theyshould produce the ollowing outcomes:

1. Maniest disagreement. Public deliberation should include periodso debate among the citizens (hereater called panelists, as in theinstance o a citizen panel) on both questions o act and more un-


24/119

21masslbp.com

damental moral issues. The absence o such a clash would suggestexcessive consensus-seeking among citizens who surely have genuinedierences in experiences and values.

2. Supermajorities. Deliberative groups should be able to work

through their dierences and oten reach broad agreement whenassessing initiatives. Narrow majority views should sometimes growinto large majorities, and minority viewpoints should sometimesprevail.

3. Inormed and coherent judgments. Citizens judgments shoulddevelop in light o the inormation presented, the views put orwardand the careul, honest discussions among participants. As a result,participants should demonstrate more inormed and coherent viewson initiative-related issues ater participating in panel discussions.Participants should be able to give reasons or their views and shouldbe able to explain the arguments underlying alternative pointso view.

One can assess these outcomes through direct observation o thedeliberative process, complemented by systematic surveys and inter-

views with participants, event moderators and otherinterested observers.

4. Assessing disagreement level. Systematic coding o an audio (orpreerably video, or ease o transcription) record o the deliberationcan establish whether disagreement took place. This can be comple-mented with interviews o participants to determine whether theysubjectively experienced such disagreements and whether there were

any potential disagreements they chose not to bring orward (i.e.,internally censored).9

5. Assessing supermajorities. This is assessed directly rom the eventrecords when ormal votes are taken by the citizen deliberators. Inall cases, it helps to survey the participants aterward, to nd out thedegree to which they (privately) supported any nal recommenda-tions.

6. Evaluating judgments. The citizens nal judgment should beevaluated by a neutral third party, as well as interested parties, toobtain their varied assessments o its soundness. In these cases (and


25/119


those where no nal judgment is reached), it is also helpul to com-bine an analysis o the deliberation with a survey o participants,so that one can assess the degree to which the inormation andperspectives provided in the event shaped citizens individual viewson the issue. In particular, post-deliberation citizens should be more

knowledgeable, have better correspondence between their views andrelevant acts, and understand the cons o whatever recommendationthey ultimately made.

Criterion 3: Infuential Conclusions/Actions

Once implemented, successul deliberative processes should showclear evidence o their infuence on the policy-making process oron the actions o the wider public. Depending on whether theyemphasize policy recommendations and/or direct action, eectivedeliberative citizen engagement should produce the ollowing results:

1. Infuential recommendations. Deliberative engagement processesshould prove to be an eective mechanism or making a policyproposal succeed or ail in light o the citizens recommendations.Specically, when a clear majority o panelists avour a particular

policy initiative, its chances o prevailing should increase, and thereverse should be true when citizens oppose a policy.

2. Eective, coordinated action. Deliberative bodies that attemptto generate change through direct action should be able to coordi-nate their post-deliberative eorts to thereby change the relevantvoluntary actions taken by the larger public, which may indirectlyspark policy changes (depending on whether the citizens action plan

involves public policy change).One can assess these outcomes through institutional, policy and

sociological analysis, which involves a history o the relevant policiesand public actions through examination o records and interviewswith ocials, activists and lobbyists.

3. Assessing infuence. This is a tricky undertaking because it is otendicult to establish baseline probabilities o policy outcomes.10 Themost eective approach is probably employing a third-party evalu-ator who combines all relevant documentation with interviews,preerably both beore and well ater a deliberative event. Long-termassessment, in particular, could determine whether the infuence o


26/119

23masslbp.com

the deliberative engagement builds (or erodes) over time.

4. Assessing action eectiveness. The same basic methods apply toaction as to policy, with the emphasis shiting rom policy analysis tosociological investigation. The latter should entail large-scale longitu-

dinal surveys to assess public behaviour.

Criterion 4: Secondary Benets

I deliberative processes are implemented and the evidence showsthat they are reaching sound and infuential judgments and/ortransorming public action, that would be enough to warrant theirwidespread adoption. Nonetheless, it is important to examine otherpotential outcomes because many deliberative civic engagement pro-grams stress the impact they have on the participants themselves, thewider public or macro-level political processes. To give a sense o therange o these secondary benets in relation to governance, herein Idescribe and suggest evaluation approaches or three: transormingpublic attitudes and habits, changing the attitudes and habits o pub-lic ocials and altering strategic political choices.11

1. Transorming public attitudes and habits. In the long term, delib-erative panels could transorm not only their participants, but alsothe larger public. Those participating in, engaged with or captivatedby the panels should report stable (or rising) levels o public trustand signs o reduced civic neglect. Voter turnout in elections mightincrease, and citizens should develop political belies (e.g., a sense opolitical sel-condence) conducive to varied orms o public partici-pation (e.g., attending public meetings, using public aairs media).

2. Changing public ocials attitudes/behaviour. Citizen deliberationcould also change how public ocials think and behave in relation tothe larger public. Government ocials could develop more avour-able views o the judgments that citizens make during deliberativeevents. Ocials should also demonstrate an awareness o the impor-tance o citizen deliberation and come to respect panel judgments. Asa sign o improved leadership, elected representatives (and agencyocials) could also begin to step away rom conventional publicopinion on initiatives in anticipation o deliberative panel judgmentsto the contrary.


27/119


3. Altering strategic political choices. In addition, the public delibera-tion could change the strategic choices made by political campaignproessionals during initiative campaigns. Panels will have succeededin transorming the electoral environment i initiative and policycampaigns begin to ocus more o their energy on addressing the

issues raised by deliberative panels (e.g., holding debates ocused onpanel issues) and incorporating deliberative panel results into cam-paign advertising. A more ar-reaching eect o the panels could bethe emergence o routine pilot-testing potential initiatives with low-cost varieties o deliberative polling, trying to understand how thepublic will view the initiative ater deliberating.12

The methods o evaluation used to assess these secondary out-comes would be as varied as the potential impacts themselves.

Measuring shits in public attitudes and habits. One can assessimpacts on participants and the larger public through surveyresearch and inspection o election records (in those countries wherevoting is not mandatory). Examples abound or what to include insuch surveys and how to assess it, but the best examples includelongitudinal assessment (to establish change over time), comparisongroups (to dierentiate deliberations impact rom those eects o

other social/political orces) and a wide variety o measures (e.g.,breaking down ecacy into multiple sub-components, such as sel-ecacy versus collective ecacy, i.e., a sense o eectiveness whenacting in a group).13

4. Measuring changes in public ocials attitudes/behaviour. Toassess changes in public ocials, survey methods likely will ail,owing to poor response rates conventionally obtained among elites.

Instead, one should assess these outcomes through interviews withpublic ocials and in-depth, longitudinal legislative and policy analy-sis that compares processes beore and ater the deliberative civicengagement, in light o other changes in the political/legislativeenvironment.

5. Detecting shits in strategic political choices. One can assessthese outcomes through interviews with public ocials, lobbyists,campaign ocials and political activists. This can prove especiallychallenging, as it requires accessing internal strategic decisions (ordocumentation thereo) within organizations whose interests maynot be well served by such investigation. I one can obtain such data,


28/119

25masslbp.com

however, it is possible to detect signs o the deliberative processexerting its infuence. For instance, policy initiatives that ail to passmuster in trial runs (i.e., in the mock deliberative polls describedabove) are subsequently withdrawn; this can indicate that antici-pation o the eventual deliberative citizen engagement process is

causing more careul vetting o the proposals such a group might putbeore policy-makers and the general public.

Conclusion: Integrating Evaluative Methods

Table 2 summarizes the preceding discussion and breaks downevaluative methods into two columns. The rst describes a basicevaluation those methods most readily deployed on a modestbudget and within a narrower time rame. The second column aug-ments these basic methods with additional assessment tools, whichmay require more labour, money and time. Whether the evaluationrequires more than a basic method depends on the resources andgoals, but it is important to recognize the limitations o the basicevaluation approaches in terms o their reliability and validity.

In conclusion, it is important to consider how one integratesthese various evaluation metrics. That is, how does one move rom

separate assessments o each criterion (or sub-component) to anoverall evaluation o the deliberative citizen engagement process as awhole? This depends, again, on ones conception o the project, butthe ollowing approach will apply to many such programs.

Each o the three elements o design integrity count as pass-ailelements, and a subpar evaluation on any one o these yields a nega-tive summary evaluation o the entire process. That is, i any aspecto the design ailed to meet basic standards or integrity, the other

outcomes o the process are all suspect.The three elements o sound deliberation and judgment should

be viewed as parts o a coherent whole, such that one arrives at asingle assessment o deliberation/judgment in light o each element.The third o these might be most important (i.e., the coherence andsoundness o the groups judgments), but this should be weighed byhow rich the disagreement was and how eectively the group couldmove toward a supermajority. Outstanding perormance on twoo these criteria might obviate lower perormance on another, butoutright ailure on either the rst (disagreement) or third (quality ojudgment) should yield an overall assessment o program ailure.


29/119


Criterion Basic Evaluation Additional Evaluative Method

Sound Deliberation and Judgment

Unbiased

framing

Third-party document

inspection prior todeliberation

Inspection by interested parties

Third-party inspection ofprocedure instructionsand direct observation ofprocess

Inspection by both third-partyand interested parties before,during and after deliberation

Compare citizenparticipant demographics

with census data

Conduct detailed survey ofcitizens and target population to

check for differences in bothcensus and attitudinal variables

Direct inspection ofdeliberation for signs ofdisagreement

Survey participants to judgetheir subjective experience ofdisagreement and check forself-censorship of potentialdisagreements

Check final vote tallies Survey participants to learn theirdegree of private support fortheir public recommendations

Third-party assessmentof the citizens finaljudgment in light ofavailable information

Inspection by interested partiesand survey of participantsrelevant knowledge/perspective

Third-party documentinspection prior todeliberation

Inspection by interested parties

Process quality

Representative

Manifestdisagreement

Supermajorities

Informed andcoherentjudgments

Influentialconclusions/actions

Third-party assessmentof policy impact

Take longer-term assessmentsto capture gradual/eventualimpact (or detect erosion ofinfluence)

Influentialrecommenda-tions

Third-party assessmentof impact on publicbehaviour

Inclusion of large-scale,longitudinal population surveys

Effective,coordinatedaction

Design Integrity


30/119

27masslbp.com

The infuential conclusions/actions criteria are dierent in that someprograms will emphasize only one or even neither o these cri-teria. All deliberative citizen engagement programs, however, shouldorient toward one or the other to at least a degree, lest deliberationbecome seen as merely discussion, disconnected rom action. Eventhen, poor perormance on a programs relevant infuence criteriondoes not impugn the entire exercise; rather, it suggests the need orimproving the component o the program that leverages infuence.

Finally, assessment o secondary benets stands apart romthese other criteria in that program success may not require evidenceo these impacts. I a program is well designed, deliberative and

infuential, these become bonus eects, not strictly necessary orjustiying the citizen engagement program per se. In the long run,however, these secondary benets could be o tremendous value ora public and its political culture. A more engaged public, legitimateinstitutions and responsible, deliberative politics could dramaticallyincrease the capacity or shared governance and public action and,ultimately, yield much better public policy. Such potential impactsshould be assessed, or evidence o these changes could increase theestimated value o deliberative citizen engagement, thereby warrant-ing the time and resource expense it requires.

Criterion

Secondary Benefits

Basic Evaluation Additional Evaluative Method

Transforming

public attitudesand habits

Post-deliberation survey

of participants

Longitudinal survey (and

analysis of voting records) forboth deliberation participantsand wider public

Interviews with publicofficials

Legislative and institutionalpolicy analysis

Third-party assessmentof changing politicalclimate

Intensive interviews andstrategic document analysiswithin policy-relevantinterest/advocacy groups

Changingpublic officialsattitudes/behaviour

Altering strategicpolitical choices


31/119

28The Canadian Experience

This chapter evaluates the eectiveness o community engagementand public participation eorts in Regional Health Authorities

(RHAs) in Canada. An extensive review o the literature examiningpublic engagement theories and evaluative methodologies and inter-views with health care and other social services practitioners helpedestablish some common themes and directions in public engagement.Though not meant to be comprehensive, case studies are includedthat represent a more detailed investigation o particular publicengagement processes.

Because health care delivery is a provincial responsibility, RHAs

across Canada operate independently o one another. Even RHAswithin the same province have signicant dierences in approach andmethodologies. This is largely due to the act that the legislation andpolicies that are used by provinces to mandate community engage-ment or RHAs are oten very general in their requirements andreporting rameworks.

The literature and interviews suggest that in regions where a

commitment to public participation in a health care delivery systemis being implemented in a meaningul way, there is a growing con-sensus about what constitutes an eective engagement, in terms oboth processes and outcomes. Outcomes are increasingly measuredby population health and patient-centred metrics such as increasedinvolvement in health programs and client satisaction with healthcare service. Common themes or eective processes in publicengagement are oten dened using terms such as respect, diversity,

meaningul participation, accountability and equity.A growing body o work in the eld o community engage-

ment evaluation categorizes three types o evaluation summative,ormative and developmental. Summative evaluations measure

Brian Eng, Wellesley Institute

The Canadian Experience:Observations and Lessons rom theCanadian Health Sector


32/119

29masslbp.com

the outcomes that are the end result o engagement exercises, suchas uptake o services, client satisaction, better health outcomes.Formative evaluations measure progress in achieving process-oriented goals, such as participant satisaction with the process,appropriate inormation, meaningul dialogue, adequate repre-

sentation o community diversity. Developmental evaluation isan emerging eld that attempts to measure change and is used inengagement eorts that are working on complex problems in whichoutcomes tend to be unpredictable and goals, purposes, contexts andso on may change as the engagement process develops. An exampleis attempting to measure the relative impact o particular processes inmoving toward change is the way systems unction.

Summative evaluation has been used extensively in the healthcare system even prior to the advent o community engagementprocesses. The metrics and indicators are well understood, and thereis a general consensus about how to apply them.

Formative evaluation is less well developed in public engagementactivities in the health sector. There are some promising initiativesin this area and a growing body o practice in the health care andother social services sector around evaluation o processes such as

stakeholder analysis, comprehensible inormation dissemination andaccountability. It is important to note that ormative evaluations takeplace regularly in engagement processes, albeit on an ad hoc andoten personal basis. Practitioners oten point to the need or morerigorous methodologies that can transcend personal and institutionalbias (both positive and negative) and where the tools and outcomesare comparable across engagement activities.

Developmental evaluation is very much an emerging eld andis not being addressed in any signicant way in the health sector.Engagement in the health sector is still driven mostly by the needso health authorities to deliver health care programs. While thedenitions o health care programs has expanded beyond access tomedical care and now includes programs to encourage healthy liv-ing, the engagement processes are still driven by predetermined goalsand anticipated outcomes that lend themselves to summative and

ormative analysis. However, as health authorities begin to grapplewith their role in looking at the broader social determinants o healthand the systemic changes required to make progress in these areas,developmental evaluation will become increasingly important in


33/119


engagement processes.1The case studies in this chapter will help to illuminate some o

these concepts and provide some indicators o common themes andchallenges.

Drivers or change

Since the mid-1990s, governments in Canada have been devolvingthe responsibility or allocating resources in social services deliveryto community levels. The theory behind this action is that local com-munities are better at determining their social service needs thancentralized bureaucracies. The relative merit o this approach is stillopen to debate, but the process is well advanced in many areas.

With this devolution o responsibility has come a greateremphasis on nding ways to involve the community members,sector organizations and other partners in discussions and decision-making about resources allocation. Public participation, communityconsultation and community engagement have become importantcornerstones in the delivery o social services.

Traditionally, community engagement generally meant inorm-ing the public about available services and encouraging them to

use those services. This is now being augmented by processes suchas roundtables, advisory committees, town halls and open orums,where community members and stakeholders receive inormation,discuss options, and sometimes have decision-making power aboutthe nature o services.

This is certainly the case in the health sector. Provincial govern-ments across Canada are moving away rom the model o directly

unding service delivery organizations such as community healthcentres (CHCs) and hospitals. They are establishing Regional HealthAuthorities and local networks responsible or developing compre-hensive service delivery programs. In many cases, the provincialgovernment requires these regional bodies to have a communityengagement strategy to inorm and guide the development o theirplans.

The traditional method o public participation in health care

delivery was through involvement on the boards o directors osel-governing institutions such as community health centres andhospitals. These bodies were unded directly by the government andmay or may not have had other community engagement strategies to


34/119

31masslbp.com

develop their programs. A report commissioned by the Associationo Ontario Health Centres in 2006 indicated that this model waschanging signicantly.

Across Canada, there has been a general shit in how publicparticipation in health care is being carried out. Community gov-

ernance has been shiting away rom direct democracy o locallyelected community boards toward engagement through variousother mechanisms such as inormation sharing and consultation, andby the establishment o community advisory committees, councils orgroups. From the perspective o governments, devolved authority toregional structures and the encouragement o citizen participation inplanning and priority setting through these various means is seen as

moving health care closer to communities. But locally governed com-munity health organizations and individual community memberssee these trends as a movement toward more remote and centralizedgovernance. Although community engagement is being promoted asa means to involve citizens in health care planning, empowerment olocal citizens (including the most vulnerable populations) achievedthrough local community governance may be declining.2

Given the caution expressed in this report, it is clear how impor-

tant it is to have an evaluation ramework o community engagementto determine whether the goals o community involvement andempowerment are being met.

Cape Breton

The Cape Breton District Health Authority (CBDHA) is one o ninehealth authorities in Nova Scotia. It is primarily a rural catchment

with several medium-sized towns and an urban centre o 25,000people in Sidney.The Nova Scotia Health Authorities Act requires that each

District Health Authority (DHA) establish Community HealthBoards (CHBs) to serve as the eyes and ears o the community.This is a primary vehicle or public consultation and participationin Cape Breton. There are six CHBs in the Cape Breton DHA thatcover a range o rural and urban catchments. Board members are

recruited through local advertising and word o mouth.CHBs provide advice to their health authorities about the needs

o their community. Depending on the CHBs internal inrastructurecapabilities, this advice can result rom internal board discussions or


35/119


public consultations such as community orums.As well, CHBs distribute unding to community agencies imple-

menting DHA programs. They are required to report back to theDHA about the use o those unds and the impact on health out-comes in their community.

The Cape Breton District Health Authority (CBDHA) pro-vides a summative evaluation across the district o this inormationand makes it available to the public on its website. This evaluationincludes a description o key indicators o health and liestyle out-comes. A progress report is updated each year.3

One problematic process is the recruitment and retention oCHB members. An ad hoc evaluation using an inormal survey

method and conversations with existing and past board membersidentied key issues such as volunteer burnout, transportation andunderstanding the role o CHB members.

CHBs communicate with one another within each region andacross the province through a council o chairs, enabling them toevaluate their experiences with other engagement processes that havesimilar mandates. In act, a new sta position Community HealthBoard coordinator was recently established at the Cape Breton

District Health Authority ater members heard about the eective-ness o similar stang components in other health authorities in theprovince.

The CBDHA interviewee indicated that ormative evaluation oCHB work is on the agenda. At the moment, there are limited toolsin use, but additional tools are being contemplated. The intervieweeexpects that the major challenges will be nancial i the implementa-

tion is resource-intensive. It is assumed that there will be signicantsupport rom senior sta and policy-makers at the health authority.According to the interviewee, They are not araid o change.

Winnipeg Regional Health Authority

The Winnipeg Regional Health Authority (WRHA) has a thoroughand well-researched Community Development Framework.4 Theramework promotes the regions organizational development and

acilitates networking, inter-sectoral collaboration, public participa-tion initiatives and local area development. The ramework denescommunity, establishes a participation model and outlines methodso public participation.


36/119

33masslbp.com

The WRHA oers the ollowing rationale or having a compre-hensive community development model.

Community development empowers people to have morecontrol over the decisions that infuence their own health and thehealth o their community through increasing personal control over

their own health behaviour change and by addressing the underly-ing health determinants such as poverty, housing, or environmentalthreats. The concept o empowerment is ocused on achieving equityin health and increased public participation in health program deci-sion-making.

The public participation process involves six advisory councilsand dozens o place-based and program-specic working groups.

Rigorous evaluation determines whether the advisory councils arerepresentative o the communitys diversity. Potential advisory coun-cil members are interviewed and asked to provide inormation aboutthemselves, including sel-identication with minority or marginal-ized groups. Advisory council members are chosen with a view tothe overall makeup o the councils being representative o the com-munity. There is also an evaluation ramework or the work o thecouncils based on sel-reported perceptions o the work and processes

o the group, as well as indicators o community interest in partici-pating in the work o the councils. These reports are rolled up andanalyzed by sta to the health authority and reported to the WHRAboard.

The ramework also outlines extensive evaluation tools or thevarious working groups. These tools are used by acilitators to moni-tor the progress o the group as it denes goals, begins to understand

issues, assesses participation and so on. This process has been inplace or a short time only and, at present, is used primarily bythe acilitators to track and rene processes. However, the WRHAinterviewee elt that standardizing the evaluation will help to create amore general picture o the eectiveness o the community develop-ment model once the resources are in place to do a ull-scale rollup othe inormation. The interviewee also indicated that one o the pur-poses o the standardization was to nd evaluative tools that t with

models that are more prevalent and understood in the health carecommunity. In other words, to nd models and tools that evaluatequalitative actors in a culture that is more used to and adept at usingsummative, quantitative tools.


37/119


An example o an innovative WRHA development is in theirwork with other government departments and social service agen-cies. The models and evaluative tools along with human resourcesare now being made available as a part o an inter-sectoral collabora-tion, and it is hoped that comparing outcomes across sectors will help

to rene the evaluative tools and the methods and processes o publicparticipation.

Saskatoon Regional Health

In many ways, Saskatchewan has one o the longest histories opublic participation in health care. When one sta member at theSaskatoon Health Region was asked, Why do you engage the pub-

lic? they answered, Because we are the province o medicare. TheSaskatoon Regional Health interviewee reinorced this point by say-ing that the region is itsel as a steward o public unding and elt anobligation to engage the public or eective resource allocation.

There is a legislative requirement in Saskatchewan to have pub-lic input into the health care system. However, the province doesnot monitor this except as a complaint-based system. The SaskatoonHealth Region has a clear Community Development Framework.5

Like many others, it has identied reasons or community engage-ment that include encouraging community participation in health,ocusing on the creation o healthier communities and expanding theunderstanding o actors that sustain health o communities. It alsoidenties principles and methods.

According to the interviewee, there is very little rigorous evalu-ation o the program as a whole. However, there is evaluation o

particular initiatives, primarily driven by external unders. Theexample cited was o an Aboriginal partnership in which they willbe hiring an evaluator to assess the outcomes o the program and, ashow well important values such as respect, equity, integrity and soon are being incorporated and to help drive the engagement process.

The Saskatoon Health Region realized that its advisory councilsystem seemed to be foundering, so it surveyed advisory councilmembers about what was working and what was not and asked

or suggestions about what could be improved. Implementing theserecommendations, however, has been hampered by a recent changeo government that has resulted in a transition period while thegovernment examines new bureaucratic processes. The interviewee


38/119

35masslbp.com

recalled that this kind o delay has happened in the past and oeredthe opinion that ormative evaluations o this kind are most use-ul when they can be applied over time since they are intended tomeasure progress rather than outcomes. The evaluations becomeineective when there is constant restructuring and repositioning

o ormats and methodologies that are the result o bureaucratic orpolitical imperatives rather than o the evaluations themselves.

The Saskatoon Health Region comprises both the urban cen-tre o Saskatoon and surrounding rural areas. The intervieweementioned one rural community engagement ocused on usingdiscretionary public unding to preserve a local acute care acility.The questions were asked: How do you evaluate that engagement

outcome against the act that it seems relatively clear that the sameamount o resources put into programs or healthier living, coupledwith programs to reimburse transportation costs or medical carein the nearby urban municipality, will result in better health out-comes or the community as a whole? Are people not receiving theright inormation? Are they not assimilating it? In other words thequestion that is being asked is how to evaluate whether or not theengagement is meaningul.

The Saskatoon Health Region is keen on developing a bet-ter understanding o community engagement and how it can beeectively evaluated. To this end, it is involved with a Regional Inter-sectoral Committee that has commissioned an evaluation o publicparticipation in social service delivery in the province.

Vancouver Coastal Health

Vancouver Coastal Health (VCH) includes 25% o the populationo British Columbia in an area that covers the city o Vancouver, itssuburbs and as ar along the coast as Powell River. VCH has veCommunity Health Advisory Committees: three are geographicallybased, one works with the Aboriginal community and, in 2006,a Palliative Care Community Reerence Committee was established.As well, VCH delivers numerous project-based engagement exercisesintended to provide advice to the health authority on program

development.

In 2006, VCH hired a consultant to help sta develop evalua-tion methodologies and tools. The resulting ramework identied the


39/119


purpose o the evaluation as to assess the practice (or process) ocarrying out community engagement processes and the impact (out-come) that they have on VCH decision making.6 The rameworkalso established a set o questions or consideration.

A set o surveys to be completed by participants and project

leaders at the conclusion o selected community engagement consul-tations was created as an evaluation tool. Follow-up surveys are alsosent out. Several dierent templates have been developed or use indierent contexts.

The process evaluation is intended to evaluate a broad range oprocess-oriented questions, including demographics and participantmotivation and satisaction. Sample questions include Did partici-

pants eel like their opinions matter to the organization? and Didthey have enough inormation to be able to contribute ully?The outcome evaluations have their own particular nuance in

that they are not evaluating program outcomes in the traditionalsense. Instead, they are being used to determine the value o commu-nity engagement (CE) processes to particular projects. For example,questions are asked to determine whether project leaders eel thatthe CE process was useul to their project outcomes and what moti-

vates project leaders to integrate a CE process into their projectwork plan.

The intention o the VCH sta has been to compile thesereports as part o their yearly reporting process. However, as withmany other engagement processes, it is a challenge to nd the statime and other resources to properly systemize, collate and analyzethese reports. As the VCH interviewee put it, there is a constant

need to do, do, do. Time spent in evaluative work by departmentsta can be perceived as time taken away rom delivering the com-munity engagement program.

Nevertheless, VCH sta do spend time in ad hoc evaluationand process analysis as part o their ongoing work. This is a naturalpart o the day-to-day discussions and collaboration among sta.The interviewee did oer the observation that a more systematicapproach to collating the learnings rom these evaluations might

prove valuable in bridging the culture cap between the communitydevelopment unctions and the health care delivery unctions. Butshe oered the caution that any evaluative process needs to be used,analyzed and understood in terms o the contexts in which engage-


40/119

37masslbp.com

ments take place. Any attempt to overly systemize evaluations in anattempt to make them applicable across wide varieties o engagementprocesses will likely result in ailure due to receiving insucientinormation or the potential or misinterpreting inormation withoutknowing the context.

St. James Town Initiative

One o the many things to be learned about evaluation rom otherelds o community engagement practice in the health care sector isthe burgeoning eld o Community-Based Research (CBR). Duringthe development o CBR projects, ormative, process-based evalua-tions are oten used to dene and oten rene the purpose, scope and

methodologies o the project.The St. James Town Initiative o the Wellesley Institute, orexample, was envisioned by its initiators as a research project thatwould look at neighbourhood actors in newcomer health outcomes.It is well known that new immigrants to Canada tend to be healthierthan the general population but that their health outcomes declineover time. There is a general understanding about the drivers o thisphenomenon in terms o lower incomes and more dicult access to

culturally appropriate health care. The Wellesley Institute wantedto examine neighbourhood actors in a distinct geographic area thathas a high immigrant population. North St. James Town in Torontowas a likely candidate. The methodology initially envisaged was aqualitative study that would ollow a select group o individuals andamilies or a period o time using periodic surveys, ocus groups andso on. There was also an intention to do a quantitative study.

Sta at the Wellesley Institute began to work with community-based organizations in St. James Town to recruit people or thequalitative study and to seek input on the indicators. Almost imme-diately, the community members engaged in an ad hoc ormativeevaluation to determine whether they wanted to participate. TheWellesley Institute was told that i the sole purpose o the engage-ment was to learn more, then St. James Town community memberswerent interested, concluding: Our community has been studied to

death. What we need is action.

The Wellesley Institute repurposed the engagement to includean action component, evaluating the initial methodologies to deter-


41/119


mine whether they could result in action. They concluded that theyhad to introduce some new, more participatory research componentsthat would allow community members to dene early in the pro-cess some areas o action that might be pursued as the engagementproceeded. As a result, the rst methodologies employed in the quali-

tative research were Photo Voice, Community Mapping and ConceptMapping. This allowed participants to make an early identication oneighbourhood actors that might be actionable.

Common Themes

Some common themes have emerged rom the research.First, the health care system is relatively adept at using sum-

mative, quantitative evaluations to assess the impact o communityengagement and public participation processes. It can look at pro-gram evaluations, uptake models and so on to determine whetherhealth outcomes are improving as a result o engagement. Theindicators and metrics are reasonably well understood and agreedupon. The evaluations have expanded beyond just access to medicalservices and now include other elements o population health.

Second, it is clear that ad hoc ormative evaluations take place

during many engagement processes. These are natural and otenunintended evaluations that occur because acilitators or partici-pants want to know that their eorts are eective and valuable.Interviewees and other practitioners have indicated that more rig-orous ormative evaluations would be valuable in assessing andimproving engagement processes.

Third, as the health care sector moves into more complex

engagement processes that examine the larger systemic issues thatimpact health outcomes, more complex evaluative tools will need tobe developed and used to assess the outcomes and processes o theengagements.

Challenges

There are a number o challenges in evaluating community engage-

ment and public participation in the health sector, particularly romthe view o ormative or developmental evaluation.

One challenge is that common denitions or ormulationso various key terms and components do not exist. What is pub-


42/119

39masslbp.com

lic participation in one place is analogous to civic engagement inanother and community development somewhere else. There areno common denitions o key community, stakeholder, user and soon. Goals and principles o community engagement vary rom oneprocess to another. This occurs not only in the heath care sector but

across the entire eld o community engagement and development.Advances have been made in this area,7 but urther opportunitiesexist to rene our denitions by collaborating across processes. Onebenet would be to move the ormative evaluations rom the ad hocand sel-reported methods to more rigorous methods in which theoutcomes and learnings can be more easily shared across jurisdic-tions and sectors.

Another challenge is that the more rigorous evaluations tend tobe resource-intensive. The long-term benets o diverting resources,particularly in the health care sector, rom immediate problem-solving into evaluation have to be clearly articulated. As well, thelevel o unding required can sometimes be unpredictable at the starto an engagement process. This is problematic or unders, whorequire predictable costs.

On the cultural ront, large institutions oten resist new tools and

techniques. Without a champion o innovation within the institution,there is a tendency to use what appears to have worked in the past.There is also a resistance to evaluating or outcomes that appear tobe outside the scope o the engagement process. While strengtheningtrust in democratic processes is regularly identied by practitionersas a probable and desirable outcome o a successul engagement,there is very little work being done by institutions to identiy this and

to try to monitor outcomes. Community capacity building is alsooten cited as a goal o community development and engagementprocesses. But it is hard to nd any indication that this is being moni-tored or evaluated in any signicant way.

There is also some resistance on the part o the participants inthe process. Many o the new evaluative tools are highly resource-intensive in terms o both the time required rom participants in theengagement and rom acilitators and evaluators. It also requires a

substantial buy-in rom the participants. This can be particularly di-cult i some individuals eel they have time constraints. There is alsoa tendency to want to roll up the sleeves and work on the problem,and people may not appreciate the value o regular breathing spaces


43/119


to evaluate where they are and where they might be going.

Conclusions

The health care system in Canada has undergone undamentalrestructuring in the last 20 years. There has been a greater emphasis

placed on public participation in developing and allocating resources,setting priorities and creating programs. The degree to which this callhas been taken up has varied considerably across the country.

It is important to note that most health care authorities andinstitutions are in the business o allocating resources, developingpolicies and creating programs that address particular health prob-lems. However, it must be recognized that many public engagement

processes (particularly those that genuinely involve grassroots com-munity members) will tend to move beyond these limitations toaddress the larger issues o public policy, community values, equityand accountability, thereore:

Use a mix o evaluative methods

There are dozens o possible community engagement tools available,ranging rom public awareness programs to online surveys, ocus

groups, public meetings, citizens assemblies, advisory councils andcommunity health boards. People will participate in these orumsby a variety o means. They may volunteer to be part o a process.They may be randomly selected. They may be elected rom theircommunities or appointed by institutions.

Because o this variety o methodology, a variety o evaluationtools must be available to determine whether the engagements are

eective. Traditional summative methods may be most useul orgroups working on particular programs. More ormative evaluationsmethods may be necessary to determine the eectiveness o thoseengagements working to dene values and priorities. Developmentalevaluations will be most useul in complex collaborative eorts seek-ing systemic change.

Plan or evaluation

Evaluation needs to be addressed at the beginning o the process.Particularly with ormative and developmental evaluation; theengagement participants should be involved in the planning processrom the start. Eective community engagement is an iterative,


44/119

41masslbp.com

evolving process, and regular evaluation o the process itsel will leadto outcomes that have greater impact.

Plan or dierent orms o evaluation or dierent types oengagement and even or dierent stages in particular engagements.Focus groups that are empowered to develop new programs may

need specic data sets and benet rom summative orms o evalu-ation. Citizens assembly engagements may need to evaluate thequality o inormation being provided to them as they go throughtheir process. Advisory councils may need to evaluate whether theyare representative o the community they speak or or work with.

Be fexible

Use the outcomes o evaluation to rethink priorities, directions andmethods being used. The impression that original goals arent beingreached shouldnt be seen as ailure. The engagement process maybe raising new questions, problems and solutions that need to beexplored urther. Be ready to use dierent evaluative processes asneeds and directions change.

Learn, share and collaborate

Good practices, resources and even talent related to evaluative meth-ods should be shared, both within the health care sector and withthe broader social service and community development sector. Thisis particularly true in the use o ormative and developmental evalu-ation. These types o evaluation tend to ocus on processes that areexperiential and oten values-laden. The more we can begin to iden-tiy common denitions and methodologies, the more we will be able

to compare outcomes across dierent engagement processes. Whileit is important to be able to nuance and adapt denitions and tools todierent contexts, it is also important to develop shared understand-ings o broad-stroke concepts. This will lead to evaluative methodsand tools that will produce more meaningul learnings and improvethe eectiveness o community engagement and public participation.


45/119

42


46/119

43masslbp.com

You will be involved. The local NHS [National Health Service] willinvolve patients, carers, the public and other key partners. Those

aected by proposed changes will have the chance to have their sayand oer their contribution. NHS organizations will work openlyand collaboratively.

NHS Next Stage Review: Leading Local Change, May 2008

This chapter examines existing indicators used in England to mea-sure meaningul engagement and public condence in the health caresector.1 It also examines the challenges that exist in measuring patient

and public involvement in health and lessons that can be learnedrom the British experience. Recommendations and conclusions aredrawn rom a comprehensive literature review as well as interviewswith ve leading engagement specialists in the health eld, carriedout in late 2008. These experts are spearheading many o the initia-tives to improve the quality o health engagement evaluation andassessment.

Traditionally, the National Health Service (NHS) has aordedvery limited involvement to patients and the public. NHS cultureassumed that patients are passive recipients o health care servicesand that their needs were best anticipated and managed through top-down structures that let little room or meaningul consultation ordevolved autonomy. Today, that culture is changing. The desire tosustain public condence and a newound recognition o the exper-tise and experience o patients is driving calls or innovation and

reorm.

Drivers or change

Three reasons explain the shit toward greater patient and public

The English Experience: Evaluating Patientand Public Engagement in Health

Edward Anderson and Emily Fellen, Involve


47/119

44The English Experience

engagement. First, across the wider public sector, the appetite amongcivil servants or involving citizens in service design and decision-making exercises has grown. This appetite stems rom the practicaldesire to improve the quality and responsiveness o public services,and serves a secondary interest by connecting the experience o pub-

lic services to a broader democratic agenda. The drive or public andpatient engagement in health care should be viewed within this widercontext.

Second, a new ocus on prevention and behavioural change hasemerged within the health sector in order to meet the needs o anaging population, the rising cost o medical interventions and thegrowth o complex chronic disease. Practitioners and health admin-

istrators have come to view enhanced engagement as an importanttool or encouraging behavioural change and healthy living.Third, a series o widely publicized incidents required NHS

administrators to restore public trust in the health service and its gov-ernance. An early example was the 2000 Kennedy Inquiry into thehigh mortality rate at a childrens heart surgery in Bristol. Among itsrecommendations, the inquirys report included no ewer than10 recommendations aimed directly at the issue o public involve-

ment and empowerment.

Key developments and challenges

Legislative and regulatory changes have also had a powerul eect.Since 2001, all health bodies in the United Kingdom have beenrequired to consult and involve patients in service planning andoperation. As o October 2008, these requirements were extended to

include relevant communities in assessing commissioning decisions.The new duties, embedded in section 242 o the National HealthService Act 2008, extend patient and public engagement rom theservice delivery arena to strategic decision-making. Increasingly,patients and the public are being viewed as ull operational andstrategic partners in the provision o British health care.

Another key change has been the introduction o new structuresor engagement, such as Local Involvement Networks (LINks) set up

in early 2008. Designed to provide a link between citizens and ser-vices, these networks are a vehicle or ongoing engagement. LINkshave been established alongside local councils. Unlike previous NHSengagement structures that relied on articial or unrecognized health


48/119

45masslbp.com

boundaries, LINks serve existing and well-established communitiesand integrate easily with local governance structures.

But despite new on-the-ground inrastructure such as LINks anda strong legislative mandate, the NHS aces diculties in implement-ing section 242 o the 2008 NHS Act. Little headway has been made

in gauging the uptake or success o these recent requirements.Some existing indicators show troubling developments. For

example, public trust in NHS is in decline; the British SocialAttitudes Survey 2006 highlighted that just 12% o respondents had agreat deal o trust that the NHS would spend money wisely or thebenet o citizens.2 The challenge o accurately and meaningullymeasuring the impact and quality o public engagement is not unique

to the health arena, although it does ace its own particular obstacles.First, any indicators o successul engagement need to take intoaccount the variety o organizations working within the health eld;good engagement practices at a commissioning organization arelikely to look quite dierent rom the engagement practices amongservice providers. There is also a considerable dierence betweenengaging patients in their own care and engaging members o thepublic in policy-making or planning.

Second, the NHS has traditionally been driven by quantitativetargets and indicators based on clinical outcomes. These hard tar-gets are not well suited or easily adapted to the qualitative and highlycontextual work o patient and public engagement. Not surprisingly,many researchers and organizations have struggled to develop robustindicators that can measure meaningul engagement outcomes.

Moreover, most eorts at engagement do not typically yield

immediately identiable and causal clinical improvements. Theseshortcomings can uel a clash among clinicians, administrators andproponents o engagement who argue that scarce resources shouldbe spent on these activities.

A third challenge is the relative novelty o engagement or theNHS and the constant change o NHS policies, structures and priori-ties, which has hampered attempts to evaluate and refect on healthengagement structures. This bureaucratic churn has muddied the

waters and made it diculty to properly evaluate the ecacy omany programs and initiatives.

Together, these actors explain why a denitive rameworkor assessing engagement has yet to be developed in the United


49/119

46The English Experience

Kingdom, though eorts by the Healthcare Commission, theDepartment o Health and the National Centre or Health andClinical Excellence are each underway.

The evaluation o health engagement in England

Drawing on interviews with individuals at the oreront o theengagement in the health arena and in particular with individualsinvolved in setting up rameworks or evaluating engagement thissection examines the extent and quality o evaluations o Patient andPublic Engagement (PPE) activity in England. It then looks at spe-cic case studies and outlines several o the methods or evaluatingpublic engagement in health currently in use in England. It concludes

by elaborating on the themes emerging rom the literature reviewand interviews, and discusses what needs to happen in order

Mass Engaging Report

Documents

Transcript of Mass Engaging Report