April 25, 2012

Patient-Centered Outcomes Research Institute

Special Public Board of Governors Webinar and Teleconference

Special Public Board of Governors Webinar and TeleconferenceApril 25, 2012

Eugene Washington, MD, MSc, Chair, PCORI Board of Governors

Joe Selby, MD, MPH, Executive Director, PCORI

Welcome and Introduction

Special Public Board of Governors Webinar and TeleconferenceApril 25, 2012

Christine Goertz, DC, PhD, Member, PCORI Board of Governors

PCORI Pilot Project Program Award Recommendations

4

The Process

Step One: Determine Universe

Step Two: Determine Approach

Step Three: Develop Core Slates

Step Four: Analyze Balance

Step Five: Make Final Recommendation

Step Six: Approve

The decision was made to eliminate all applications with a score above 3.0 from consideration for further funding. This yielded a potential fundable pool of 80 applications.

The criteria to be used for balancing were refined and two options for selecting a base slate of applications were proposed.

Staff applied the two selection process options to the top 80 applications, yielding two possible sets of fundable applications (one with 37 selected; the other with 50).

Using the balancing criteria staff analyzed the balance of each of the two possible sets to determine if balance was achieved or if additional balancing activities would be required.

The Selection Committee reviewed the two possible sets of fundable applications to determine if balancing was needed. The group decided on two options for the Board and selected one for recommendation.

The Board of Governors will vote on the recommendation.

This process was completed using only generalized information regarding the applicant. No Selection Committee member was aware of the name of the PI or the applicant organization during this process.

5

The selection comm. considered all potential options for selecting the slate of fundable applications and decided to go with an option that would use priority score first and then ensure the top two applications were selected in each review group. All selection committee members were blinded as to the names and affiliations of the applicants.

• Percentiles (as noted on the NIH web site) are calculated usually for study sections that have had at least three meetings – traditionally, the last three rounds during a year.  Thus, percentiles for the Pilot Projects Program do not have the same meaning as NIH percentiles. They reflect the application’s ranking within a single study section meeting.

• Priority scoring is likely to be the most reliable measure across all groups since all reviewers had the same training regarding how to score.

• We don’t know (since there is no history to the review group) if a “poor” score is due to a review group that just scores “harsher” or the grants they had were just not very “good”.

Rationale for Selection Method

6

Rationale (cont’d)

• NIH attempted to assign applications to reviewers based on areas of interest, though it was difficult given the large number of applicants and PI’s did not necessarily self-assign correctly as well as listing more than more area. Thus, the assignment was imperfect and an applicant that was not funded (if we used percentiles as the primary selection criteria) could argue s/he might have been funded had s/he been assigned “correctly”.

• However, we did want to take into consideration the potential that some groups could be harsher and give some weight to ranking within the review group – so, we decided to also pick up the top two applications in each review group.

7

Balancing Criteria

Balancing Criteria

Definition Operationalization Source

Area of Interest* The eight areas of interest listed in the PFA

At least 2 unique applications but no more than 50% in any one area

At least two members of the Selection Committee read each application abstract and determined the primary areas of interest.

Population*Defined as addressing 1) Specific ethnic or cultural group, 2) disabled populations, 3) children, and 4) elderly populations.

At least 1 application in each of the four categories

PCORI staff reviewed abstracts to determine if the application had a clear focus on one or more of the four populations.

Condition* The disease or condition used to demonstrate the approach

No more than 25% in any major category within the final slate.

Staff reviewed the abstracts to categorize conditions addressed, if any

Stakeholder/Patient Involvement*

The average score given by the three reviewers for this criterion in the initial merit review group

Any application added to the core slate will have a score of 1 or 2 (except methods focused applications)

The average score from the IRG review was used.

Geography Geographic location of institutional affiliation

Will be reported but not used for balance.

The state or country on the application face page for the PI.

Method The innovation of the research method

Only to be used if balance is needed within the area of interest related to methodologies (8).

The methodology committee would make the determination, if needed.

PI DisciplineCategorization of PI qualifications based on primary area of expertise

Insufficient data to use

PI Seniority Whether the PI has received federal funding Insufficient data to use

Eight potential criteria were discussed and refined. The top four were proposed as the most appropriate for balancing.

Option 1: Balancing

Option 1 takes the applications with a priority score of 25 or better and ensures

at least the top two applications from each panel are included. This option

yields 50 applications and has only one balancing issue:

Issue: Area of Interest 4--There are no applications within this slate with that

designation as a primary area of interest. Developing, refining, testing, and/or evaluating methods to identify gaps in CE knowledge such as tools for the ongoing collection and assessment of gaps as perceived by patients and providers. Of special interest are gaps that are particularly relevant to vulnerable populations, including but not limited to, low-income populations; underserved minorities; children; the elderly; women; and people with disabilities, chronic, rare, and/or multiple medical conditions.

Balancing Recommendation: There are no applications within the top 80

that have this designation as a primary area of interest. The committee

recommends we not balance on this area and propose it be a focus in one of

the new PFA’s or use a contract mechanism to address.

Option 2: Balancing

To remain as close to 40 grants (number approved by the BoG), option 2 takes applications with a priority score of 24 or better and ensured the inclusion of the top application in each review panel. This resulted in 37 applications.

1. Issue: Area of interest 4— This is the same issue encountered under Option 1.

Recommendation: This issue should be handled in the same way as in Option 1.

2. Issue: Area of interest 1—There is only 1 application within the slate that addresses this area.

Developing, testing, refining, and/or evaluating new or existing methods (qualitative and quantitative) and approaches that can inform the process of establishing and updating national priorities for the conduct of patient-centered outcomes research (PCOR). This may include research prioritization approaches (such as Value of Information (VOI), burden of illness, peer review/expert opinion/Delphi approaches) or methods for incorporating the perspectives of patients or other stakeholders into the development of national priorities.

Recommendation: To add an additional application with a primary area of interest of 1, the committee looked within those applications with a priority score of 25. Within that group there is one application with a primary area of interest of 1.

Discussion and Voting

Selection Committee Recommendation

• The committee recommends the Board vote for Option 1 and that we not try to balance on Area 4.

Special Public Board of Governors Webinar and TeleconferenceApril 25, 2012

Board Discussion and Consideration of PCORI Pilot Project Program Award Recommendations

Special Public Board of Governors Webinar and TeleconferenceApril 25, 2012

Leah Hole-Curry, JD, Member, PCORI Board of Governors

Draft National Priorities for Research and Research Agenda

Review Genesis of National Priorities and Research Agenda

Review Public Comment Process

Share Public Comment Findings

Recommend PCORI Response

Board Vote on Recommended Changes

Objectives for Today

13

14

The What and Why of National Priorities and Research Agenda

Mandated in the legislation (including Public Comment period)

Pre-requisite for releasing funding announcements

Preliminary roadmap for PCORI research activities

Envisioned as a living document

Developed Framework to be used for refining priorities and for determining Research Agenda and funding announcements

Reviewed initial stakeholder input advising us to not “reinvent the wheel”

Development of Draft National Priorities and Research Agenda

15

Initial Stakeholder feedback

Reviewed prior CER frameworks (e.g., IOM, FCCCER, National Priorities Partnership, and NQF)

Environmental scan of existing priorities and

criteria

Identified broad priorities from prior frameworks and the statutory criteria for PCORI

Candidate priorities and

criteria identified

Framework to inter-relate

Priorities and Criteria

Establishing PCORI’s First National Priorities for Research and Initial Research Agenda

9Criteria outlined

by law

5Draft priorities

proposed

Corresponding agenda drafted

Public input received and

evaluated

Priorities and agenda revised and approved

First primary funding

announcements issued

Aug-Dec 2011 Jan-Apr 2012 May 2012

16

Patient and Caregiver Focus Groups

17

Columbus, OH (Dec. 7)• Parents of children with ADD/ADHD

(caregivers)• Patients with Mental Health

conditions• Patients who survived Cancer

• Insured and uninsured• Age: 30-70+

Phoenix, AZ (Dec. 8)• Patients with Respiratory Disease

(chronic bronchitis, emphysema)• Hispanic patients with mix of

chronic conditions• Patients with Heart Disease

• Insured and uninsured• Age: 21-69

Atlanta, GA (Nov. 21)• Patients with Diabetes• Caregivers to Alzheimer’s patients• Patients with Chronic Pain

• Insured and uninsured• Age: 21-75+

Baltimore, MD (Nov. 9)• Patients with Arthritis• Parents of children with Pediatric

Asthma (caregivers)• Patients with mix of chronic

conditions• Insured and uninsured• Age: 21-69

Receiving Public Feedback on the National Priorities and Research Agenda

18

• Clinician Focus Groups

o Clinician focus groups took place in cities across the USo Philadelphiao Birminghamo Californiao Chicago

o Four groups of physicians and four groups of nurses

o Behavioral Health Professionals

o Physician Assistants

A commitment to public engagement

19

Formal 53-day Public Comment Period

• Nearly 500 comments received through website, e-mail or postal mail

• All comments will be posted at pcori.org

Additional Forums

• National Patient and Stakeholder Dialogue

• Patient, caregiver and clinician focus groups

• Individual meetings with diverse mix of stakeholders

Public Comment Yielded Diverse Array of Input and Feedback

Broad range of comment sources

301 Web survey (64%) 116 Email/ letter (24%)

57 Stakeholder event (12%)47

4 To

tal C

omm

ents

20

Those Who Commented Said They Understand the National Priorities…

Addressing Disparities Accelerating PCOR and Methodological Research

Assessment of Options Improving Healthcare Systems Communications and Dissemination Research

“How well do you understand the National Priorities?”

21

Rigorous, Systematic Review and Analysis of Comments

Computer algorithm to identify key terminology

Each narrative comment reviewed and analyzed by 3 people

Stakeholder comments aggregated to 15 key themes

Themes compared to National Priorities and Research Agenda to identify gaps

Gaps reviewed to determine options for PCORI response to themes

22

Options for Response

Change Language Within the Research Agenda

Embed in PCORI Operations and Processes

Address in Summary Document

Future Consideration

23

Recommends that PCORI choose a specific condition, disease area, or other issues in the Research Agenda and National Priorities

PCORI has proposed a condition-neutral Research Agenda and has introduced specificity through its comparative nature and emphasis on patient centeredness. While future funding announcements may specify conditions, the overall mission of PCORI is not served by excluding any conditions if there is compelling reason for a patient centered, comparative clinical effectiveness study. We added language to elaborate on our position.

Recommends that PCORI partner with organizations and stakeholders to carry out its mission

PCORI is committed to efficient use of its research investments. Where appropriate, PCORI will partner with other organizations after a transparent decision-making process and consideration of conflicts of interest. This theme impacts PCORI processes, rather than funding subjects, so no specific language changes were made to the document.

Recommends greater focus on the patient, with particular attention to methods of engagement

PCORI has fully endorsed and appreciates the centrality of patient engagement to its mission. The National Priorities and Research Agenda reflect the patient centered focus of PCORI and include many of the themes from the public comments. Language has been added to the Agenda to specifically reflect the need for study of self care and to more clearly define personalized medicine.

Recommends a greater focus on care coordination PCORI appreciates the need to study care coordination and has expanded the language in the Research Agenda to reflect its importance.

Recommends funding towards improving patient and provider health literacy and education

Improving communication between patient and provider is one of the five PCORI National Priorities for Research. Language has been added to the Research Agenda to reflect the importance of health literacy to achieving this goal.

PCORI Response to Key Themes-1

24

Recommends funding for and use of health IT infrastructure, networks, tools and patient data acquisition efforts in and outside the practice setting

The foundation for performing comparative clinical effectiveness requires substantial health IT and data infrastructure. The National Priorities and Research Agenda contain substantial language about this infrastructure. Therefore, no additional language was added to the document. PCORI will support reusable infrastructure for comparative clinical effectiveness research.

Recommends that PCORI pay greater attention to the role of caregivers and other stakeholders in the patient decision making process

PCORI appreciates the role of caregivers in patient centered care and has mentioned them in the document and included studies of caregiving in the Research Agenda. Therefore, no additional language was added to the document.

Recommends that PCORI pay greater attention to access to care, including the social and environmental determinants that determine access and use of care

Access to care is a key issue for patients. Language has been added in both the comparative assessment and the healthcare systems Research Agenda topics to include the comparative study of access as a determinant of health.

Recommends that PCORI provide greater rationale and transparency in the public comment, grants, and research evaluation processes, as well as the performance measurement process for PCORI as a whole

PCORI is committed to fully transparent processes as it works towards achieving its mission. The Research Agenda articulates the ongoing engagement that will occur continuously as PCORI evolves and funds research. Therefore, no additional changes were made to the document. PCORI intends to roll out a comprehensive communications and engagement plan that will clearly define when and how stakeholders can provide input into PCORI decision making.

Recommends that PCORI's research and funding should impact the practice setting, with particular attention to patient and provider behavioral change needed to obtain true shared decision making

PCORI is fully committed to the idea that its research should improve decision making and help patients at the point of care. Language has been added to the section “Establishing the Scope of the Research Agenda” to emphasize the importance of using the evidence developed through PCORI research to change the way medicine is practiced.

PCORI Response to Key Themes-2

25

Recommends that PCORI place stronger emphasis on patients with multiple conditions, especially chronic conditions

PCORI understands the difficulty of managing multiple chronic conditions when most evidence is generated in trials that exclude these patients. Language has been added to emphasis this in the Research Agenda.

Recommends that PCORI study new and expanded roles for allied health professionals

PCORI recognizes the diverse health professionals involved in patient centered care. In the Research Agenda, the description of allied health professionals has been expanded to be more inclusive of all of potential members of a health care team.

Recommends paying attention to international models PCORI recognizes the significant achievements of many countries in developing the methods and practices of patient engaged comparative clinical effectiveness research that may inform investigators as they seek PCORI funding. As this is not central to PCORI research, no change is proposed to the priorities or agenda.

Recommends exploring novel methods to obtain patient centered focus

PCORI supports the approach of exploring innovative methods for focusing on the patient. The fundamental basis of PCOR, however, is the science of evidence-based medicine. PCORI will support and promote approaches that seek rigorous, scientific results; therefore no changes were made to the document.

Recommends that PCORI study rare diseases PCORI recognizes the challenges faced in studying rare diseases. In the Research Agenda, language about rare disease has been expanded.

PCORI Response to Key Themes-3

26

Discussion

Vote on Changes Outlined in Slides 13-15

PCORI Stakeholders

Thank you for your thoughtful input into the first version of the PCORI National Priorities for Research and Research Agenda

PCORI Program Development Committee

Thank you for all your hard work in the development and refinement of these documents

The detailed Summary Document of changes to the National Priorities and Research Agenda will be posted on pcori.org by April 30, 2012

Thanks

29

April 25, 2012

Patient-Centered Outcomes Research Institute

Special Public Board of Governors Webinar and Teleconference