Panel of Eminent Persons_Lessons learned for the OSCE from its engagement in Ukraine
Engagement in PCORnet 1.0 · Methods •Surveyedall35 PCORnet networks to identify engagement...
Transcript of Engagement in PCORnet 1.0 · Methods •Surveyedall35 PCORnet networks to identify engagement...
Engagement in PCORnet 1.0:Results of the Engagement Assessment Project
Sharon F. Terry, MA
Andrea Goodman, MSW, MPH
Meanwhile, at today’s Feline Engagement Committee meeting, decisions were made on how to increase feline recruitment.
Project Background
PCORnet has prioritized engagement as a key function in transforming healthcare research
However, there was little investment in capturing and scientifically assessing engagement activities in PCORnet
PCRF is definitive on this topic – engagement is a critical tenant
GA CC set out to capture, review and publish the resources, best practices, and lessons learned from PCORnet and its networks
3
Project Overview and Goals Capture and assess engagement activities across the Network to learn and synthesize:
§ Engagement methods, tools, and models from across the Network
§ Successes, challenges, and lessons learned from PCORnet engagement activities
§ Gaps and opportunities for systematic, effective, and measurable engagement
§ Indicators of successes and challenges § A recommended framework for evaluation to assess impact§ Overall knowledge and capacity to execute and evaluate
engagement
4
Methods• Surveyed all 35 PCORnet networks to identify engagement goals,
strategies, activities, challenges, and lessons learned
• Interviewed representatives from each network after the initial survey to
probe deeper around gaps, challenges, and specific engagement
practices
• Analyzed survey data using Qualtrics and Microsoft Excel to produce
descriptive and inferential statistics
• Coded, categorized, and organized qualitative data into emerging
themes using NVivo, a qualitative data analysis computer software
package
• Data included interviews, network reporting, Commons resources,
past presentations, and prior open-ended survey results
5
A Snapshot of Respondents100% response rate for surveys and interviews (13 CDRNs, 20 PPRNs, and 2 HPRNs)
64 individuals participated in the interviews
Diversity of participant types:
6
Who Networks Are EngagingNetworks reported involving a variety of stakeholders across some aspects of network operations and research activities
All networks are engaging patients/participants (100%) and researchers (100%), the majority are engaging clinicians (97%) and systems leaders (94%), and some have also said they involved payers (47%)
7
Engagement Strategies and Definitions
Most networks (34/35) reported developing their own strategy for
engagement
60% use a formal framework and 86% use some tools or
resources
100% of networks use a formal definition of engagement (many
report using the PCORI or PCORnet definition, or both)
8
Different Approaches
VS
PPRNs
CDRNs
Reported Success of Engagement
0 2 4 6 8 10
Patients
Clinicians
Organizationalor Systems Leaders
Researchers
Health Plans/ Payers
Reported Success of Engagement by Stakeholder Group
PPRNs CDRNs HPRNs
Factors for network self-rating:
� How far they have come in a somewhat new research paradigm� Comparison against what they thought other networks were achieving� Experiences and satisfaction of the stakeholders � Variety of stakeholders and variety of activities they were engaged in� How they “felt” is was going, many sharing pride in the work they did� Failures, shortcomings, and plans for engagement that were never realized
Reported Success of Engagement (Cont.)Most Successfully Engaged Stakeholder Group by Network Type
11
Least Successfully Engaged Stakeholder Group by Network Type
65% of PPRNs reported they were most successful at engaging patients/participants, compared to 38% of CDRNs
Overall, networks report being least successful at engaging health plans/payers (60%) and clinicians (31%)
Rating the Success of Engagement Methods
“The type of outreach needs to be aligned with the medium… there are certain things that you can do in person that you can’t do digitally or do well digitally that you couldn’t do in person… it’s not a perfect one-to-one.”
-PCORnet CDRN
Themes of Engagement Methods: Reasons for Success
13
In-Person(e.g. conference or meeting)
• Builds trust and relationships
• Promotes active participation
• Helps engage ancillary partners with project
• Reach people that may not have been reached via other methods
• Supports creative thinking and brainstorming
Digital Tools (e.g. portals or surveys)
•More efficient in terms of time and resources
•Get answers to specific questions or input on topics
•Get candid feedback about stakeholder experience and engagement approaches
•More easily reaches a large number of stakeholders
•Allows for more representative group to be involved
•Increases visibility
Social Media(e.g. Facebook or Twitter)
•A good place for community listening
•Useful for recruitment and network registration
•Easy for community to control
•Useful for dissemination
•Can engage large groups and target specific communities or groups of individuals
•People are already comfortable and engaged with platform .
•Easy to maintain regular contact
•Customized emails work to let stakeholders know you are paying attention to them
•Can get statistics about open rates and clicks
•Works well with established members of network
•Easy to share seamlessly across network members
•Get input quickly--especially useful for busy people
Phone Calls
•Helpful to reach underserved populations with limited internet access
•Used to build initial connection
•Regular calls are a good way to share updates and get feedback.
•Stakeholder preference: "People want phone calls."
•Good alternative to in-person meetings for collaboration
Themes of Engagement Methods: Challenges & Drawbacks
14
In-Person(e.g. conference or meeting)
• Time and
resource
intensive
• Can limit
consistency of
stakeholders/
individuals
involved
• Hard for people
across the
country to
gather regularly
Digital Tools (e.g. portals or surveys)
• Can add
additional cost
• Limited to those
with internet access
• Competing for
attention-- may
not be a priority
to engage, surveys may
have limited
response rates
Social Media(e.g. Facebook or Twitter)
•A lot of competition
for attention
•Limited exposure of messaging without
paid "advertising"
•Time to develop a social media
campaign
•Can be confusing if
network does not
have strong brand in community
•Needs to be
monitored and policed
•May only reach a small percentage of those sent messages
•A lot of competition for attention
•Cannot reach those who don't actively use email or have no internet access
•Feels less personal, limited opportunity to build trust and relationships
Phone Calls
• Time intensive
(compared to low
touch methods)
• Not as engaging as in-person
meetings
• Length and
frequency of
phone meetings can be tricky
Stakeholder Involvement Across Phases
15
Phases of Research
# of
Net
wor
ks
Rep
ortin
g In
volv
emen
t
“…there have been times where our researchers just don’t realize that they actually are going to need patient or clinician stakeholder involvement.”
- PCORnet CDRN
Levels of Involvement
Challenges of EngagementAll networks reported some challenges associated with the development or implementation of engagement, most frequently:
Time – More time is needed to engage effectively
Resources – Lack of staff and monetary resources that quality engagement requires
Diversity - Lack of diversity, especially within the patient/participant stakeholder group
Navigating Roles - Challenges related to stakeholders navigating their own role and that of other stakeholder groups
Collaboration - Frustrations related to a general lack of stakeholder involvement or lack of collaboration
17
Engagement Impact
No significant difference between ratings by PPRNs and CDRNsNetworks report that engagement efforts resulted in stakeholders influencing many areas of the network and research activities (see table) Stakeholder impact was harder for networks to clearly define § Examples of some impacts reported
include:• Intrapersonal changes (i.e., how
individuals approach the work) • Structural changes (i.e., how the
infrastructure was set up to support the work)
18
Networks report stakeholders influencing:
culture network decisions and utilization
dissemination of information recruitment
execution of research or projects
relationships and partnerships
data sharing funding and applications
Overall, networks rated the impact of engagement on network operations at 4.2 with a range of 2-5.
Measuring Engagement For many groups, engagement was a binary variable – success was determined by the presence of engagement
Networks rely heavily on PCORnet milestones as a benchmark of successful engagement
Examples of quantitative metrics reported include: the number of stakeholders involved, the number of engagement activities, and/or the number of resources or outreach efforts
Most networks reported measuring engagement monthly or quarterly
19
Ways Networks Measure Engagement
PCORnet Logic Model
20
Inpu
ts
- Resources (e.g. time, staff, funding)- Expectations;
stakeholder or network specific and those set
by PCORnet milestones
- Strategy developed to navigate goals and
needs of network and research
- Leadership (e.g. network directors,
national leaders and support)
Act
iviti
es- Advisory group and governance
committee meetings- Stakeholder Trainings
- Opinion Surveys-Listening sessions, community engagement studios, and focus
groups-Stakeholder interviews - Network and Registry
Recruitment- Email Updates
- Conference and Workgroup meetings
- Webinars- Social Media Outreach
- Inclusion of stakeholders in research cycle (e.g. using online tools to prioritize study topics and
research questions) - Inclusion of stakeholders in network infrastructure (e.g.
participant/patient Investigators and governance committees)
Out
puts
- # of stakeholders involved- # of meetings held
- # of patients trained- # of stakeholders reached via
email, social media, etc.- # of voices captured to inform
research- # of stakeholders in governance
- # of people recruited into networks, registries, or research- # of developed engagement resources-and infrastructure
# of stakeholder-informed research questions and proposals - # of studies with results shared
in accessible language - # of meaningful relationships
developed
Impa
cts Observed Impacts
- Increase in project funding- Added value to research teams- More patient-centered research methods and approaches- Preliminary shift in culture of researchAnticipated Short-Term Impacts*- Increase in patient recruitment and retention- Increase in research relevant to patients and stakeholdersAnticipated Long-Term Impacts*- Increase in research translated into care improvements- Improved health outcomes
Compiled Engagement Resources
Type of Resource Example # Identified
# Networks Contributing
Engagement Frameworks & Strategies Engagement Key Driver Diagram
10 11
Evaluation Models and Tools Member Survey 18 13
Methods for Engaging Stakeholders Recruitment Guide 24 16
Outreach and Comms Tools Acronym Guide 29 14
Stakeholder Roles & Governance Engagement Committee Charter
40 19
Training & Orientation Overview PowerPoint; Trainings
6 5
21
Network members mentioned 127 engagement resources
Full directory of resources available on Commons: PCORnet Engagement Assessment Project: Findings and Recommendations
Conclusions
Networks have developed and implemented a variety of methods for engaging stakeholders in their network-building and research activities
Few are formally measuring or evaluating effectiveness or success of these methods and activities in a way that looks at nuance, quality, or impact
Networks largely see engagement as a binary variable, measured by the presence or absence of it
While anecdotes can be vivid and compelling, for now, the most common quantitative measures that PCORnet networks use to report success are process measures (e.g. the number of stakeholders involved in a project, the number of meetings held with stakeholders, the number of stakeholders in governance, etc.)
22
ConclusionsNetworks often cite needing additional funding, staffing, and resources to adequately support engagement efforts
Networks articulate a desire to learn from each other and wish there were more opportunities to collaborate and share best practices, lessons learned, and resources
However, networks are also hesitant to formally share materials, approaches, or tools until they are perfected, published, vetted and branded by their community
While many networks discuss the importance of engaging a diverse and representative group of stakeholders, few offer attempted solutions or strategiesto overcome this challenge§ Networks have not consistently captured or reported on specific approaches § There are few data showing what is or is not successful in this area
23
Recommendations: Standards for Stakeholder Engagement
A diverse and project-appropriate group of stakeholders should be involved in all network and project governance; including holding positions with the highest decision-making authority
All projects should have continuous patient/participant leadershipthroughout all stages of research and involve a diverse and representative group of appropriate stakeholders throughout the research cycle
Patient/participant and stakeholder engagement should be appropriately planned and budgeted for at the beginning of a research project
24
Recommendations: Impact of Approaches and Methods
On the Network level, evidence-informed approaches should be indexed, catalogued, tested, and modified based on post-implementation evaluation
Use evidence-informed approaches to build engagement strategy and conduct regular assessments to ensure the needs of both the scientific and stakeholder communities are being met
Gather evidence to document the value engagement brings to specific aspects of the research cycle (e.g. recruitment, retention, and participant satisfaction)
Embed evaluation activities across network initiatives§ Document if and how activities have led to an increase in patient
recruitment and retention§ Document if and how activities have led to an increase in research
relevant to patients/participants and other stakeholders
25
Recommendations: Expectations for Collaboration and Growth
A shift in culture is required on the Network level, including both carrots and sticks to align around “people-centric” methods and measures
Networks should be expected to share resources, collaborate on solutions, and implement promising approaches
Spaces (both online and in-person) should be resourced for these collaboration efforts
Assess representativeness of Network and national-level stakeholder groups, the depth and breadth of reach, and the diversity of teams and partnerships
Provide opportunities for entire teams to participate in continued training in Patient Centered Outcomes Research principles, patient engagement, and team dynamics
26
Recommendations: Outreach, Information Sharing, and Transparency
Conduct expansive outreach to patient/participant community that includes multi-modal methods of communication and leverages innovative partnerships
Return study results to stakeholders in a timely and appropriate manner, using channels indicated as most valued by the stakeholder community.
Increase transparency and invest in approaches that increase participant awareness of the research ecosystem including research agendas, priorities, processes, and data use.
27
Going Forward: Raising the Bar for Engagement
Build consensus around engagement definitions, strategies, and minimum requirementsPromote a culture of collaboration with wider sharing of tools and resourcesStandardize expectations around the level of involvement of stakeholders throughout the study cycleImplement universal metrics and a standard approach to evaluating quality and impact of engagementLead the development and advancement of engagement science
“…[engagement] has to start from the ground up
rather than coming at it after
the fact [to] fix things that aren’t
working.”
- PCORnet CDRN
Questions
?!
30
31