1

Writing Writing Proposals, Proposals,

Getting Getting Reviews, Reviews,

and and PerseveringPerseveringMing Tai-Seale, PhD, MPHMing Tai-Seale, PhD, MPH

School of Rural Public HealthSchool of Rural Public Health

2

AcknowledgementAcknowledgement

Enola Proctor, PhDEnola Proctor, PhDWashington Univ at S.L.Washington Univ at S.L.

Kenneth Wells, MDKenneth Wells, MDUCLAUCLA

3

The AgendaThe Agenda

Life of a proposalLife of a proposal Scientific review: who, where, howScientific review: who, where, how Critical areas for improvementCritical areas for improvement Summary statementSummary statement Finding your agent: working with Finding your agent: working with

program staffprogram staff

4

5

3Overall StrategiesOverall Strategies

Think First, Second, and ThirdThink First, Second, and Third Get Feedback at all Stages (3 times) Get Feedback at all Stages (3 times) Develop and Follow Timeline for Develop and Follow Timeline for

SubmitSubmit Plan on 3 Major RewritesPlan on 3 Major Rewrites Talk to Agency Staff Early, 3 TimesTalk to Agency Staff Early, 3 Times After a Good Idea and Opportunity, After a Good Idea and Opportunity,

It’s Methods, Methods, MethodsIt’s Methods, Methods, Methods-From Ken Wells-From Ken Wells

6

Make a list of 5 project ideas that Make a list of 5 project ideas that you would want to work on for yearsyou would want to work on for years

Prioritize and sketch out 1-3 ideasPrioritize and sketch out 1-3 ideas Get feedback from senior Get feedback from senior

researchers, clinicians, or patients researchers, clinicians, or patients

and community membersand community members

Finding IdeasFinding Ideas

7

Have a Conceptual Have a Conceptual FrameworkFramework

Have a Conceptual Have a Conceptual FrameworkFramework

8

Make a list of 5 project ideas that Make a list of 5 project ideas that you would want to work on for yearsyou would want to work on for years

Prioritize and sketch out 1-3 ideasPrioritize and sketch out 1-3 ideas Get feedback from senior Get feedback from senior

researchers, clinicians, or patients researchers, clinicians, or patients and community membersand community members

Having IdeasHaving Ideas

9

Shaping the IdeaShaping the Idea Based on literature, agency Based on literature, agency

priorities, identify next steps suitable priorities, identify next steps suitable to your stage of development and to your stage of development and institution capabilitiesinstitution capabilities

Identify data sources neededIdentify data sources needed Identify special opportunitiesIdentify special opportunities SENIOR CONSULTSENIOR CONSULT

10

Clarify settings and subjectsClarify settings and subjects Develop Partnership – What Develop Partnership – What

model of collaboration?model of collaboration? Consider diversity goalsConsider diversity goals

CONSULT with a statisticianCONSULT with a statistician

Develop the DesignDevelop the Design

– HypothesesHypotheses– SamplingSampling– DesignDesign

11

Developing the Design Developing the Design (cont.)(cont.)

Develop rough ballpark for budgetDevelop rough ballpark for budget SENIOR HELP NEEDEDSENIOR HELP NEEDED

Discuss concept with funding Discuss concept with funding agency, based on aim, opportunity, agency, based on aim, opportunity,

design, and likely budgetdesign, and likely budget

12

Develop Conceptual Framework Develop Conceptual Framework Identify Main VariablesIdentify Main Variables

Draft Interventions, if applicable Draft Interventions, if applicable Develop Pilot DataDevelop Pilot Data

GET SENIOR CONSULT AND GET SENIOR CONSULT AND BEG FOR $BEG FOR $

Write Aims, Background, Design Write Aims, Background, Design (Pretend you’re almost done!)(Pretend you’re almost done!)

First DraftFirst Draft

13

First DraftFirst Draft

Statistician Consult to help outline Statistician Consult to help outline main analysis and develop main analysis and develop Power Power Calculations to determine:Calculations to determine:

Is the Study Affordable? Is the Study Affordable? If Yes, PROCEED TO PARK PLACEIf Yes, PROCEED TO PARK PLACE If Not, Go Back to Start (JAIL) If Not, Go Back to Start (JAIL)

14

Second DraftSecond Draft Develop Operations plansDevelop Operations plans

team organization, data collection, timelineteam organization, data collection, timeline Detailed Budget--GET HELPDetailed Budget--GET HELP

Statistical Consult: Detailed Analysis PlanStatistical Consult: Detailed Analysis Plan Review assumptionsReview assumptions Don’t delegate blindlyDon’t delegate blindly They are YOUR hypotheses; They are YOUR hypotheses; Modify design, scope, budgetModify design, scope, budget as neededas needed

Human Subjects Section: Human Subjects Section: Consult with IRB, MentorsConsult with IRB, Mentors Plop revised draft together Plop revised draft together

15

Third DraftThird Draft Meet all agency requirementsMeet all agency requirements Develop budget justificationDevelop budget justification Highlight “value added,” pilot data, Highlight “value added,” pilot data,

fit of aims, method, and analysis to fit of aims, method, and analysis to modelmodel

AGENCY & SENIORAGENCY & SENIORREVIEWREVIEW

16

It’s Not Over YetIt’s Not Over Yet Take the feedback seriouslyTake the feedback seriously Revise the whole proposal if neededRevise the whole proposal if needed

Common problems: Not feasible (budget too Common problems: Not feasible (budget too large or scope too broad ); No pilot data; large or scope too broad ); No pilot data; Aims not specific; Background has literature Aims not specific; Background has literature but no synthesis, framework, value-added but no synthesis, framework, value-added unclear; Methods are under-developed unclear; Methods are under-developed (alternatives not considered; analysis not tied (alternatives not considered; analysis not tied to hypotheses; design flaws: wrong sample for to hypotheses; design flaws: wrong sample for aim, aim,

causal inference poor; limitations causal inference poor; limitations unaddressedunaddressed

17

18

DisciplinesDisciplines Anthropology Anthropology BiostatisticsBiostatistics EconomicsEconomics Epidemiology Epidemiology Health services Health services

researchresearch Medicine Medicine NursingNursing Organizational TheoryOrganizational Theory SociologySociology

MethodologicaMethodological Orientationsl Orientations QuantitativeQuantitative QualitativeQualitative MixedMixed

Stages in Their Stages in Their Own CareersOwn Careers Senior ScholarsSenior Scholars Emergent Emergent

scientistsscientists

Who Serve on Study Sections?

19

The ReviewThe Review

Source: Enola Source: Enola ProctorProctor

20

Reviewer chargeReviewer charge

Assess likelihood that proposed Assess likelihood that proposed research will have substantial impact research will have substantial impact on pursuit of NIH research goals:on pursuit of NIH research goals:

Advance understanding ofAdvance understanding ofBiological systemsBiological systemsImprove control of diseaseImprove control of diseaseEnhance healthEnhance health

21

Review CriteriaReview Criteria SignificanceSignificance ApproachApproach InnovationInnovation InvestigatorInvestigator EnvironmentEnvironment Priority

Populations Budget Human Subject

Protections

SummarySummaryMajor Major

strengths and strengths and weaknessesweaknesses

RecommendatiRecommendation for or on for or against fundingagainst funding

22

Significance Is the dependent variable important to

NIH objectives? Does the study extend boundaries of

current knowledge? Does the study address funding agency

priorities? Bridging Science and Services Surgeon General Surgeon General’s report IOM: quality Agency partnership: “The Road Ahead”

Can the findings inform policy or practice?

23

Approach: Aims

Is the research question fully specified?

Aims are manageable, but reflects long term agenda

Incorporates relevant concepts, variables

24

Approach: Methods Setting and context

Collaborative arrangements feasible data collection plan (sufficient subjects,

reasonable procedures, plan (sufficient subjects, reasonable procedures,good measures)

Manageable: Sufficient time for methodology and products?

Person power: enough, and the right people Analytic model = fully specified

rival explanations controlled through design or measurement of relevant variables

Procedures protect data quality Appropriate methods of analysis Sufficient power

25

Methods: Critical Areas for Methods: Critical Areas for ImprovementImprovement

Design problemDesign problem MeasurementMeasurement Choice of variablesChoice of variables Intervention/comparisonIntervention/comparison

Analysis problemAnalysis problem Choice of approachChoice of approach TechniqueTechnique TestTest

Theoretical or conceptual model or framework Theoretical or conceptual model or framework Missing, deficient, or erroneousMissing, deficient, or erroneous

26

Innovation

Original and innovative aims Novel concepts, methods,

approaches Challenges existing thinking or

approaches Advances new methods or

technologies

27

Investigator Sufficient experience to direct the project

Well trained Productive

Proposed project builds on background work investigator publications preliminary studies

Proposed work will make “next logical next logical” contribution to knowledge Is surrounded by a team who can ensure success Range of substantive and methodological expertise Multidisciplinary

Critical areas for improvement Investigator expertise deficient Needs consultants or collaborators

28

Environment Environment contributes to project

success Project capitalizes on environment,

its people, and resources and resources

Collaborations Evidence of organizational support

29

Ks Review GuidelineKs Review Guideline CandidateCandidate Career development planCareer development plan Research planResearch plan Mentor/co-mentorMentor/co-mentor Environment and institutional commitmentEnvironment and institutional commitment BudgetBudget Human subjectsHuman subjects Women/minorities/childrenWomen/minorities/children SUMMARYSUMMARY

major strengths and weaknessesmajor strengths and weaknesses Recommendation for or against fundingRecommendation for or against funding

30

Overall Review All criteria are considered when

assigning overall score Application does not have to be

equally strong in all categories in all categories

Major issue: will proposed research have major scientific impact?

31

Rating Scale in NIH Rating Scale in NIH ReviewReview

1.0 virtually flawless1.0 virtually flawless, with negligible , with negligible weaknessesweaknesses

1.51.5 extremely strongextremely strong, with a few minor , with a few minor weaknessweakness

2.02.0 very strongvery strong, but with moderate weaknesses, but with moderate weaknesses

2.5 strong2.5 strong, but with some major weaknesses , but with some major weaknesses that must be addressedthat must be addressed

3.03.0 fairfair, neutral balance of strengths and , neutral balance of strengths and weaknessesweaknesses

3.5 weak3.5 weak, but with some major strengths, but with some major strengths

32

Priority ScorePriority Score

How is the summary priority score How is the summary priority score calculated?calculated? Group average Group average Average Score times 100Average Score times 100 Equal weightEqual weight

Some are “un-scored”Some are “un-scored” What is the fundable range?What is the fundable range?

Study sections can have different normsStudy sections can have different norms When in doubt, ask the project officerWhen in doubt, ask the project officer

33

The Physical The Physical SettingSetting

34

Critical Areas for Critical Areas for Improvement in Improvement in RsRs

Design problemDesign problem MeasurementMeasurement Choice of variablesChoice of variables Intervention/comparisonIntervention/comparison

Analysis problemAnalysis problem Choice of approachChoice of approach TechniqueTechnique TestTest

35

Critical Areas for Critical Areas for Improvement in Improvement in RsRs

Weak justification for studyWeak justification for study Background and significance Background and significance

unconvincingunconvincing Literature review incompleteLiterature review incomplete

Investigator expertise deficientInvestigator expertise deficient Needs consultants or collaboratorsNeeds consultants or collaborators Theoretical or conceptual model or Theoretical or conceptual model or

framework framework Missing, deficient, or erroneous Missing, deficient, or erroneous

36

Summary Summary StatementStatement

37

How to Read the Pink SheetHow to Read the Pink Sheet

Expect the language to be Expect the language to be Frank, and Frank, and Not overly enthusiastic Not overly enthusiastic

Be emotionally detached, after the Be emotionally detached, after the initial…initial…

Talk to an experienced grant-makerTalk to an experienced grant-maker Resubmit unless you see “Resubmit unless you see “fatally flawedfatally flawed”” Do NOT resubmit right awayDo NOT resubmit right away Recruit a “cold reviewer”Recruit a “cold reviewer”

38

Take a Vacation …Take a Vacation …

39

Program StaffProgram Staff Project officer – Your AgentProject officer – Your Agent

Read your concept paper and draft Read your concept paper and draft Send it in Send it in EARLY!, and EARLY!, and OftenOften

Interpret the fundability of your priority Interpret the fundability of your priority score score

40

ResourcesResources

Instructions on how to prepare your Instructions on how to prepare your applicationapplication http://grants.nih.gov/grants/funding/phs398/section_1.htmhttp://grants.nih.gov/grants/funding/phs398/section_1.htm

ll

Follow it VERY closelyFollow it VERY closely

Panel presentation at AcademyHealth Panel presentation at AcademyHealth 20062006 http://www.academyhealth.org/2006/demhttp://www.academyhealth.org/2006/dem

ystifying2006.pdfystifying2006.pdf