Post on 11-Aug-2015
Evaluating Problem Gambling KTE Part of the OPGRC’s Moving Research Forward workshop series
February 2015
Reuse & Attribution
This work is licensed under a Creative Commons Attribution 4.0 International License.
Attribute this work as:
Bergen, A. (2015). Evaluating Problem Gambling KTE. Moving Research Forward Workshop Series, Ontario Problem Gambling Research Centre, Guelph ON.
Introductions
Why KTE evaluation?
Evaluation can help you understand:
• Need for KTE activity
• Quality of KTE activity
• Reach of KTE activity
• Impact of KTE activity
KTEà Impact?
What kind of impact?
For whom?
Under what conditions?
When?
Based on what evidence?
What counts as evidence?
Evaluation research
vs.
evaluation Research
how used?
to be published?
funded?
What are your evaluation goals?
Evaluation Continuum
How will you use the evaluation results? – What?
– So What?
– Now What?
Outputs Engagement Uptake Use Impact Causal Attributions
key questions for KTE
1. What research knowledge should be transferred?
2. To whom should research knowledge be transferred?
3. By whom should research knowledge be transferred?
4. How should research knowledge be transferred?
5. With what effect should research knowledge be transferred?
Adapted from: Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision makers? Milbank Quarterly, 81(2), 221-248.
key questions for KTE evaluation
1. What research knowledge was transferred?
2. To whom was research knowledge transferred?
3. By whom was research knowledge transferred?
4. How was research knowledge transferred?
5. With what effect was research knowledge transferred?
Adapted from: Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision makers? Milbank Quarterly, 81(2), 221-248.
Evaluation Over KTE Project Lifespan
• Before: Needs assessment
• Creation: Developmental evaluation – user experience/ usability testing
• During: Process (formative) evaluation – quality & quantity of outputs
• After: Summative (outcome) evaluation – outcomes & impacts
Evaluation Goals
• What aspects of KTE do you
want to evaluate?
• What are you going to do with
the resulting information?
Evaluation Framework
• Goals
• Values
• Approach
• Logic model/ theory of change
• Targets of change (stakeholders)
• Methods, metrics, & measures
What are your evaluation values?
More impact.
Inform Consult Involve Collaborate Empower
Less control. Slower process.
Adapted from Arnstein’s (1969) Ladder of Public Participation and the IAP2 Spectrum of Public Participation
How engaged are your stakeholders in your research & KTE activities?
What is your evaluation context?
Research Funding Organization
Program of Research A
Project A1
Product A1a
Product A1b
Project A2
Product A2a
Product A2b
Product A2c
Project A3
Product A3a
Product A3b
Program of Research B
Project B1
Product B1a
Product B1b
Project B2
Product B2a
Product B2b
Research Funding Organization
Program of Research A
Project A1
Product A1a
Product A1b
Project A2
Product A2a
Product A2b
Product A2c
Project A3
Product A3a
Product A3b
Program of Research B
Project B1
Product B1a
Product B1b
Project B2
Product B2a
Product B2b
Context
Research Funding Organization
Program of Research A
Project A1
Product A1a
Product A1b
Project A2
Product A2a
Product A2b
Product A2c
Project A3
Product A3a
Product A3b
Program of Research B
Project B1
Product B1a
Product B1b
Project B2
Product B2a
Product B2b
Evaluation Context (macro & micro)
• Micro - (time, money, resources, intended use, intended users, reporting requirements, etc…)
• Macro – (culture, leadership, evaluation, external systems, political climate, etc….)
What approach to evaluation will you
take?
What Kind of Evaluation?
Evaluation research vs.
evaluation Research
how used? to be published?
funded?
practical & theoretical considerations
Audience/ Targets of Change
q gamblers and problem gamblers
q gambling industry
q gambling researchers
q general public
q policymakers
q subsets of the general public (e.g., youth)
q
Methods
q Qualitative q Quantitative q Mixed Methods q Prospective q Retrospective q . q Participatory? q Descriptive? q RCT? q Pre-Post? q Counterfactual? q Contribution analysis? q ..
What is the theory of change?
(logical links between activities & desired outcomes)
Theory of Change
Problem or Issue
1
Needs & Assets
2 Desired Results
(outputs, outcomes)
3
Influential Factors
4
KTE Strategies
5
Assumptions
6
Adapted from Kellogg Foundation Theory of Change template http://www.wkkf.org/resource-directory/resource/2007/07/spark-theory-of-change
Theory of Change
Problem or Issue 1
Needs & Assets 2
Desired Results (outputs, outcomes)
3 Influential Factors 4
KTE Strategies 5 Assumptions 6
Adapted from Kellogg Foundation Theory of Change template http://www.wkkf.org/resource-directory/resource/2007/07/spark-theory-of-change
Another approach: logic models
Assumptions
Think about individuals, groups, systems, & assumptions
Activities Short-Term Outcomes
Intermediate Outcomes
Long-Term Outcomes
Systems & Policy Change Initiatives
• Share resources
• Create partnerships
Awareness Systems Changes
Reduced harm from gambling
Partnerships
Individual Change Initiatives
• Provide education and resources on…
• Provide environmental cues to…
Knowledge & skills
Intention & Behaviour Changes
Psychosocial
Gambling context
An example:
process-focused KTE evaluation
Final research summary created
Increased awareness of research
scope among project team members
and partners.
Increase in perceived ownership over project.
Increased clarity and relevance of summary
Increased (clear
language) communica=
on and transla=on
skills.
Document summary
candidates and full ar=cles in database.
Create agreement on process for summaries
within project team.
Hire students from across University.
Format summary using
standard template
Student writers work with faculty to
revise summary draDs
Provide training for writers and stakeholders
(staff).
Increased accessibility of summary
Increased trust of research product
(successful branding)
Ac=vi=es
Short T
erm Goals
Long Term Goals
Increase in inter-‐silo
communica=on.
Increased uptake and use of summary and original research
Increased knowledge
sharing within and outside of academia
Increased quality of summary
Increase in project team and partner collabora=on:
Industry Communi=es Policy-‐Makers Prac==oners
Staff Students Faculty
Preparation
Make research summaries available through:
Social Media Tradi=onal Media
Government Partners
Google Search University Business Support Office
Dissemination Production
Internal Collabora=on KMb
ins=tu=onal partners Collabora=on
with Researchers and Research
Users Logic model for a university-based plain language summary project
A mid-‐term evalua=on included a review of products produced, processes used (including collabora=on across the Project Team, the system for deciding on summary topics) and success of the social network strategy.
Final research summary created
Increased awareness of research
scope among project team members
and partners.
Increase in perceived ownership over project.
Increased clarity and relevance of summary
Increased (clear
language) communica=
on and transla=on
skills.
Document summary
candidates and full ar=cles in database.
Create agreement on process for summaries
within project team.
Hire students from across University.
Format summary using
standard template
Student writers work with faculty to
revise summary draDs
Provide training for writers and stakeholders
(staff).
Increased accessibility of summary
Increased trust of research product
(successful branding)
Ac=vi=es
Short T
erm Goals
Long Term Goals
Increase in inter-‐silo
communica=on.
Increased uptake and use of summary and original research
Increased knowledge
sharing within and outside of academia
Increased quality of summary
Increase in project team and partner collabora=on:
Industry Communi=es Policy-‐Makers Prac==oners
Staff Students Faculty
Preparation
Make research summaries available through:
Social Media Tradi=onal Media
Government Partners
Google Search University Business Support Office
Dissemination Production
Internal Collabora=on KMb
ins=tu=onal partners Collabora=on
with Researchers and Research
Users
1. Count & describe 2. Ask respondents
(stakeholders) to describe collab process/why various pieces were important
1. Post-training survey (quality of training; knowledge of plain language)
2. Writer & faculty survey (quality of process & outputs)
3. Useful? Easy to use? Clear? (1) focus groups of policy makers, researchers, and community partners and (2) post focus-group survey
4. Focus group/survey for stakeholders.
1. # tweets, RT, etc… 2. # followers 3. # hits & downloads
An example: descriptive KTE evaluation
Cooper, A. (2011). Knowledge Mobilization in Education A cross-case analysis of 44 research brokering organizations across Canada. Unpublished doctoral thesis, University of Toronto, Toronto, Ontario, Canada.
Cooper’s (2011) Brokering Functions of KM Strategies diagram shows possible activities & outcomes
An example: outcome-focused
KTE evaluation
Contribution analysis 1. Map a “journey” (theory of change) that links activities
to increased research uptake to outcomes
2. Identify assumptions and risks for each stage of the journey
3. Identify indicators for research uptake, use and impact
4. Collect evidence.
5. Review the map against your evidence. Identify gaps in evidence, then try to fill the gaps.
6. Write your contribution story. Explain the journey from inputs to impacts.
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18, 270-280. doi:10.1177/1356389012451663 Morton. (2013). Assessing research impact: A case study of participatory research. Centre for Research on Families and Relationships. https://www.era.lib.ed.ac.uk/handle/1842/6562
Morton, S. (2013). Assessing research impact: A case study of participatory research. Centre for Research on Families and Relationships, 66. www.crfr.ac.uk
Example of “impact journey” diagram from Sarah Morton’s work
What measures and indicators will you
use? (metrics to stories)
Outputs vs. Outcomes
Outputs Engagement Uptake Use Impact Causal Attributions
measurement time & complexity
KTE indicators for:
• Number/type of KTE products
• Ease of use/ user experience
• Timing/ relevance (meets audience needs)
• Awareness/ attitudes/ beliefs/ knowledge
• Self-reported intentions/ behaviour
• Networks/ relationships/ collaborations
• Systems/ policies/ organizational culture
Social Process, Social Impact
q Quality & quantity of relationships
q 1:1 relationships; organizational relationships
q Meetings
q Requests & referrals
q Co-produced products
q Social network analysis
q .
Group Feedback
What did you create?
+/- method: user experience testing • 10 minutes). Review the workbook on your own. Mark the margins
with a plus (+) or minus (-) to record your positive and negative reading experiences.
• (10 + 10 minutes). With a partner, review the motives for each of the +/- in the document. Ask probing questions where needed. Record your responses and note the top 3 most important positive and negative experiences.
• (15 minutes). As a small group, give feedback on top 3 +/- for each partner. What was effective/ not effective and why?
• (15 minutes). As a large group, review trends in +/- experiences for each group.
Create your own KTE evaluation framework
Evaluation Framework
• Your research • Your KTE • Evaluation goals • Values • Approach • Logic model/ theory of change • Targets of change (stakeholders) • Methods, metrics, & measures
W3
What?
So What?
Now What?
What?
What happened? What did I observe?
What stood out for me?
So What?
Why is that important? How did my understanding change?
Now What?
How will my practice change: …over the next 3 months? …over the next year?