Moving Communication Forward With Evaluation · Moving Communication Forward With Evaluation Jim...
Transcript of Moving Communication Forward With Evaluation · Moving Communication Forward With Evaluation Jim...
Moving Communication Forward With Evaluation
Jim Macnamara PhD, FAMI, CPM, FAMEC, FPRIA
Professor of Public Communication, University of Technology Sydney Visiting Professor, London School of Economics and Political Science
Measurement & evaluation Measuring communication and ‘big data’!
• Goal setting and measurement are fundamental for communication and PR
• Measuring communication outcomes is recommended versus only measuring outputs
• The effect on organizational performance can and should be measured where possible
• Measurement and evaluation require both qualitative and quantitative methods
• AVEs are not the value of communications
• Social media can and should be measured
• Measurement and evaluation should be transparent, consistent and valid
SMART objectives
Contain numbers, percentages, dates
Design evaluation at the planning stage
Be realistic
Linked to organisational objectives and goals
Achieved by a specific date
Three types of evaluation
Formative
Process
Summative
• Baseline (e.g., what is the current level of awareness)
• Precedents (have other similar programs worked in the past?)
• Pre-testing (will what we propose work?)
• CEA (is it the most cost-effective approach?)
• Insights / learning (what has been learned that can inform future strategy, policy, etc?)
• Impact (what has happened as a result of the communication?)
• Outcomes – intermediate and long-term (what attitude or behaviour change has occurred?)
• Monitoring and tracking outputs (e.g., are we reaching the audience?)
• Monitoring and tracking immediate outcomes (e.g., what response are we getting?
Information processing / communication theory
Six stages of communication (W. J. McGuire, 1968, 1969)
ComprehensionPresentation Change/actionRetentionAttention Acceptance
Advertising models and the ‘marketing/sales funnel’
The AIDA model of advertising (Strong, 1925)
AWARENESS
INTEREST
DESIRE
ACTION
Information processing / communication
Exposure
Attention
Interest and/or liking
Comprehension
Cognition, particularly cognitive elaboration (thinking about the message)
Acquiring skills or knowledge required to deal with the issue (if necessary)
Attitude change, particularly to agreement (what McGuire calls “yielding”)
Storing information in memory (retention)
Retrieving information (i.e., recall)
Deciding to act in accordance with information (intention)
Action/behaviour
Cognitive integration of behaviour (e.g., reinforcement)
Encouraging others to behave similarly – what McGuire called “proselytizing” and what is commonly regarded in modern marketing as ‘advocacy’ (McGuire, 1999, 2001)
The PII model of evaluation (Cutlip, Center, & Broom, 1985)
Macro model of evaluation (Macnamara, 1992)
Pyramid model of PR research (Macnamara, 2002, 2005)
OUTCOMES(Functional &
organisational evaluation)
OUTPUTS(Process & program
evaluation)
INPUTS(Formative
research)
Number who . change ..…..……………………………………………….. Quantitative surveys (large scale structured)
behaviour …...…………………………………………….. Sales; Voting results; Adoption rates; Observation
Number who ….. change
attitudes ……….…………………………………………… Focus groups; Surveys (targeted) (eg Customer, Employee
Number who understand messages..……………………………… Focus groups; Interviews; Complaint decline; Experiments
Number who retain messages …….………………………………… Interviews; Focus groups; Mini-surveys; Experiments
Number who consider messages ……....….. ………………………… Response mechanisms (1800, coupons); Inquiries
Number & type of messages reaching target audience ……………………… Media Content Analysis; Communication Audits
Number of messages in the media ……………..….`…………………… Media Monitoring (clippings, tapes, transcripts)
Number who received messages ………………...……………………….. Circulations; Event attendances; Web visits & downloads
Number of messages sent …………………………………..……………….. Distribution statistics; Web pages posted
Quality of message presentation …………………………………………………. Expert analysis; Peer review; Feedback; Awards
Appropriateness of message content …………………………………….………….. Feedback; Readability tests (eg. Fog, Flesch); Pre-testing
Appropriateness of the medium selected ………………………………………….…….. Case studies; Feedback; Interviews; Pre-testing (eg. PDFs)
How does target audience prefer to receive information? …………….…………..……… Academic papers; Feedback; Interviews; Focus groups
What does target audience know, think, feel? What do they need/want? ………………… Observations; Secondary data; Advisory groups; Chat rooms
& online forums; Databases (eg. Customer complaints)
or Shareholder Satisfaction); Reputation studies
Measurement Methodologies: (formal & informal)
Key Steps/Stages in Communication:
OUT-TAKES(Proposed by some as a 4th stage)
PR effectiveness yardstick (Lindenmann, 1993)
Measuring:Behaviour changeAttitude changeOpinion change
ADVANCED
Measuring:RetentionComprehensionAwarenessReception
INTERMEDIATE
Measuring:Target audiencesImpressionsMedia placements
OUTPUT
LEVEL #3
LEVEL #2
LEVEL #1
PU
BLI
C R
LEAT
ION
S EF
FEC
TIV
ENES
S YA
RD
STIC
K
OUTCOMES OUTGROWTHS OUTPUTS
Noble & Watson’s united model (Noble & Watson, 1999)
INPUT STAGEPlanning & Preparation
OUTPUT STAGEMessages & Targets
IMPACT STAGEAwareness & Information
EFFECT STAGEMotivation & Behaviour
Tactical feedback
Management feedback
Communication controlling (DPRG/GPRA, 2000; DPRG/IPV, 2009)
■ The European Commission’s (EC) Better Regulation Guidelines uses the terms:
―Inputs
―Outputs
―Results
―Impact
■ European Commission model (EC, 2015b)
European Commission (EC, 2015a, 2015b)
10 political priorities
Communication objectives
Activities Relevance Output Outtake Outcome
Recall indicators measuring the extent
to which the communication
activity is likely to have satisfied,
captured the attention of audiences, or raised
knowledge and awareness focussing
on the audience directly reached
Political match Of the communication activities undertaken with the 10 political
priorities
• Organising events • Providing
information • Working with
information networks and desks
Reach indicators measuring the extent to which extent the
communication activity is likely to
have reached the right target audience
quantitatively and qualitatively
Engagement Trust indicators measuring the extent to which the communication
activity led to either a discernible action being taken or the
desired change in the target audience’s
perception
Inputs Results
Did the activity match the 10 priorities?
Did the activity reach the target?
Was the message received? Did the activity meet the expectations of the target audience?
Did the action lead to the expected result? Did the action change the perception of the EU?
Q.
PR and communication evaluation landscape
Inputs / preparation
Activities / implementation
Outputs
Outtakes
Outcomes / outflows / outgrowths
Results / effects / impact
Program theory and program logic models
Basic program model evolved since the 1970s (Kellogg Foundation, 2004, p. 1).
Program theory and program logic models
Program logic model developed by the University of Wisconsin University Cooperative Extension Program (UWEX) (Taylor-Power & Henert, 2008, p. 5)
Program and PR evaluation models (1985–early 2000s)
Bas
ic p
rogr
am lo
gic
mo
de
l –
UW
EX (
Tayl
or-
Po
we
r &
He
ne
rt,
20
08
)
Exp
and
ed
pro
gram
logi
c m
od
el
– U
WEX
(Ta
ylo
r-P
ow
er
&
He
ne
rt, 2
00
8)
Cla
ssic
Pro
gram
Lo
gic
Mo
de
l (e
.g.,
Ke
llogg
Fo
un
dat
ion
(1
99
8/2
00
4)
Cu
tlip
, C
en
ter
& B
roo
m (
19
85
) ‘P
II m
od
el’
Mac
nam
ara’
s M
acro
mo
de
l’ (1
99
2)
Lin
de
nm
ann
’s E
ffe
ctiv
en
ess
Y
ard
stic
k (1
99
3, 1
99
7a)
IPR
A G
old
Pap
er
on
Eva
luat
ion
(1
99
4, p
p.
10
, 18
–19
)
Fair
child
(1
99
7, 2
00
1);
Fai
rch
ild
& O
’Co
nn
or,
IP
R T
oo
lkit
( 1
99
9)
Lin
de
nm
ann
Sta
nd
ard
s an
d
Gu
ide
line
s, 1
99
7b
, 19
97
c)
No
ble
& W
atso
n’s
‘U
nif
ied
M
od
el’
(19
99
)
Gru
nig
& H
on
‘re
lati
nsh
ips’
m
od
el
DP
RG
/ G
PR
A (
20
00
), D
PR
G/I
CV
(2
00
9) m
od
els
Like
ly P
erf
orm
ance
M
eas
ure
me
nt
Fram
ew
ork
(2
00
0)
Mac
nam
ara’
s ‘P
ryam
id’
mo
de
l (2
00
0, 2
00
2a,
20
02
b)
Lin
de
nm
ann
’s ‘
guid
elin
es'
(2
00
2/2
00
3)
Input Inputs Inputs Inputs Inputs Inputs (hinted)
Input Input Inputs
Activities Planning
Output Outputs
Activities Participation
Outputs Implement-ation
Outputs Outputs (basic)
Outputs Output Outputs Output Outputs Output Outputs Outputs PR Outputs
Outtake Outtakes Outtakes PR Outtakes
Outgrowths
(inter-mediate)
Outcomes Outcomes Short-term Intermediate Long-term
Outcomes Outcomes (advanced)
Outcomes Outcome Outcomes Outcomes Outcome Outcomes Outcomes PR Outcomes
Impact Impact Results Business / organization outcomes
Impact Relationships Outflow Outgrowths Business / organization outcomes
Effect
9
14
4
1
12
5 1
(Macnamara, 2016)
Theory of Change
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT
Communication • Objectives • Target audiences • Strategy
Actions
Short | Medium | Long term
The Organisation Organisation goals and objectives
Feedback loops to monitor and adjust strategy and tactics if required
Program Theory
INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES Short term Intermediate Long term OUTTAKES OUTCOMES IMPACT
Communication • Objectives • Target audiences • Strategy
Feedback loops to planning
Preparation Production Distribution /Exposure/Reception Response Effects Results Organisation/Stakeholder Impact
ORGANISATION Organisation goals and objectives
Theory of Change Program Theory
What you do (COST CENTRE) What your audience does (VALUE CENTRE)
UK Government Communication Service evaluation framework (GCS, 2015)
AMEC Integrated Evaluation Framework (AMEC, 2016)
NHS Blood and Transplant blood donation campaign
INPUTS ACTIVITIES OUTPUTS OUTCOMES OUTCOMES OUTCOMES Short term Intermediate Long term OUTTAKES OUTCOMES IMPACT
Communication • Objectives • Target audiences • Strategy
Feedback loops to planning
Preparation Production Distribution Exposure/Reception Response Effects Results Organisation/Stakeholder Impact
ORGANISATION Organisation goals and objectives
Theory of Change Program Theory
? ? ? ? ? ?
Taxonomy of evaluation – 6-stage model
Inputs Activities Outputs Outtakes Outcomes Impact
Evaluation taxonomy includes …
Low cost and even no-cost evaluation methods as well as advanced methods
Start small and work up – e.g., pilot studies
Learn from failures as well as successes – learnings inform future strategy
Visualise findings
Busy management want ‘dashboards’, simple charts
1. Not accessing data that are available Literature review ‘OP’ data Unrecognised data – look beyond numbers to consultation submissions, correspondence, etc.
2. Lack of data sharing
Breaking down ‘data siloes’ and politics – Knowledge management (KM) expertise
3. Lack of data analysis – tools and skills
Qualitative data analysis methods such as textual, content, and thematic analysis
4. Lack of resources and rigour Academic partners Interns
5. The output to impact gap – how to bridge it?
Build an evaluation bridge using a multi-stage, multi-step framework/model
Tips to overcome 5 barriers and move evaluation forward
Questions & Discussion