National Science Foundation -...

Information & Intelligent Systems (IIS) Division

Computer & Information Science & Engineering (CISE) Directorate

National Science Foundation

SCH – Smart & Connected HealthNSF 16-601

Administrative Support: Shantia Brown, Dynika Rollins

Lorraine Parker

1

Acknowledgment

• You are performing a valued public service and we appreciate your hard work prior to and during the panel meeting.

• Your participation is very important to the research and education community!

• Thanks to NSF administrative staff for their assistance with panel logistics.

2

Meeting Sign-In

3

Needed for payment.Do this every day of the panel!

First time, correct any mistakes in the spelling of your name, address, or demographic information.

Presenter

Presentation Notes

Be sure to fill out the “Meeting Sign-in” form after you log into the Fastlane Panel Review System, and do so every day of the panel. This *after* and in addition to just logging in. (See the slide.) Sign-in ensures payment Correct any mistakes in the spelling of your name, address, or demographic information When you leave, use recycle and garbage bins In case of an emergency – follow instructions for evacuating the building or shelter in place

COI: Kinds

• Institutional (“statutory”)– Current, previous (12 months) or possible future

employment– Employment of spouse or children– Paid participation in advisory committees,

honorariums, etc.• Personal (“regulatory”)

– Thesis advisor or advisee relationship– Co-author of paper, project collaborator within past

48 months– Family member or close friend

4

COI: Examples

• Close friend, relative, or business partner (forever)• Student or advisor relationship (forever)• Current or previous employee or possible future

employment at the institution (12 months) • Payment (e.g., honorarium) from institution or

individual (12 months)• Collaborator on paper (48 months)• Co-editor of a journal, compendium, or conference

proceedings (24 months)• Person with whom you have had a dispute

5

COI: What Everyone Must Do

• Turn in NSF Conflict-of-Interest form– Sign the form to certify that you were briefed on COIs

• Talk over possible conflicts with NSF program officer

• Declare actual and perceived conflicts• If you have a conflict, leave room during

discussion• Sign additional COI form, for NIH, at the end of

the panel meeting, before leaving

6

Allegations

• Suspicion/allegation of academic dishonesty or plagerism in a proposal– discuss ASAP with PO – not with panel– will be referred to the OIG– may pose COI issue – discuss with PO

7

Confidentiality• Proposals contain sensitive information, and are not in the public

domain• Do not

– disclose identities of reviewers– disclose identities of people associated with proposals (PI, Co-PIs,

Consultants, etc.)– discuss proposals or results outside the panel room, or without the

presence of an NSF staff member– mention names of other reviewers in reviews or Panel Summary– copy, distribute or quote from proposals

• You can indicate (e.g., on a resume) that you served NSF on a review panel – just don’t identify which panel(s)

• Panel makes recommendations, not decisions. You can see the list of funded proposals using Award Search http://www.nsf.gov/awardsearch/

8

Presenter

Presentation Notes

Be cautious in elevators and other places outside the panel room about discussing panel business

http://www.nsf.gov/awardsearch/

SCH Program Overview

• A cross-cutting program in Computer and Information Science and Engineering (CISE) Directorate.http://www.nsf.gov/pubs/2016/nsf16601/nsf16601.htm

• Supports collaborations of CISE, ENG and SBE research• Integrative Projects:

– ≤ $2,000,000 total cost, up to 4 years– multi-disciplinary team– at least two of CISE, ENG, and SBE– Collaboration/Coordination Plan required

9

http://www.nsf.gov/pubs/2016/nsf16601/nsf16601.htm

SCH Program Scope• Goal = develop solutions that would transform our health or

healthcare• The work to be funded by this solicitation

– Must address a key health problem– Must make a fundamental contribution to engineering, computer and

information sciences, or social, behavioral and economic sciences.• Research team must include members with appropriate and

demonstrable expertise in the major areas involved in the work.

• Activities should complement rather than duplicate the core programs of the NSF directorates and the research efforts supported by other agencies such as the NIH, Agency for Healthcare Research and Quality, and NIST.

• SCH does support evaluation, but not support clinical trials.

10

NSF Merit Review Elements1. What is the potential for the proposed activity to

a. Advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and

b. Benefit society or advance desired societal outcomes (Broader Impacts)?2. To what extent do the proposed activities suggest and explore creative,

original, or potentially transformative concepts?3. Is the plan for carrying out the proposed activities well-reasoned, well-

organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?

4. How well qualified is the individual, team, or organization to conduct the proposed activities?

5. Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?

11

Consider in top-down order.Apply to broader impact activities as well as research.

NEW!

Presenter

Presentation Notes

FROM GPG: “societal outcomes” including, but not limited to: full participation of women, persons with disabilities, and underrepresented minorities in STEM; improved STEM education and educator development; increased public scientific literacy and public engagement; improved well-being of individuals in society; development of a diverse, globally competitive STEM workforce; increased partnerships between academia, industry, and others; improved national security; increased economic competitiveness; and enhanced infrastructure for research and education. SOME NICE NOTES FROM TED: As an NSF proposal reviewer, you are not reviewing a paper for a conference or journal, grading a student paper, or debugging a program. You are advising the US Government on how to invest taxpayer research dollars. So, try think like a venture capitalist. We can all find fault with anything, and computer programmers and engineers are especially good at finding bugs. That is not the purpose here. Every proposal has some defects. We don't want to fund projects just because the proposals are well written and have all the right pieces. We want to fund research that is innovative, and is likely to have a positive impact. We can afford to take some risks, if the potential payoff for success is high. So, a potentially "transformative" idea may deserve funding even if the proposal has not worked out all the details or dotted all the "i"s. We also fund some "incremental" work, if it is needed to bring earlier work to fruition, but we are not just interested in paying people to write research papers. Remember that you are evaluating the proposal, not the proposer. While the qualifications and prior work of the investigators are important considerations in judging whether they can successfully execute the project, the reputation of the PI should not be considered as a primary factor in evaluating the proposal. FROM KEN: in particular for CyberSEES, stimulating a new research community...

What are Broader Impacts?• Implicit (new knowledge, field, benefits to society …)• Explicit (technology transfer, results dissemination …)• Integration of Research and Education

– Development of curriculum and supporting materials– Student involvement in emerging research and technology – Postdoctoral training (plan required if the proposal requests

funding for one or more postdocs)– Data management (plan required)

• Broadening participation of underrepresented groups– Computer Science education, computer systems workforce– Gender, ethnicity, disability, geographic, etc.

• CANNOT JUST BE THE HEALTH IMPACT

12

SCH-Specific Review Criteria • Participation of health professionals, client groups, etc.• Infrastructure planning & software sharing• Survivability – commercialization, translation to

practice• Impact – on health and/or healthcare• Collaboration & Management – team integration• Education & Training – integrated with research• Potential for novel functionalities and features, user-

friendliness, interoperability, scaling, extensibility, durability, patient-centeredness, machine learning, and direct application to advancing the nation's health

13

Consider these when evaluating merits of proposals that are in-scope for the program.

Presenter

Presentation Notes

Additional Solicitation Specific Review Criteria In general, group and multi-institutional proposals requiring significant resources to generate effects are expected to show their potential for novel functionalities and features, user-friendliness, interoperability, scaling, extensibility, and durability, patient-centeredness, machine learning, and direct application to advancing the nation's health. These proposals should, as appropriate, include active participation from healthcare professionals, client groups, technology vendors, and potential commercial enterprises or application beneficiaries. The proposals will also be evaluated based on: Infrastructure planning and software sharing: Per NSF policy, the dissemination plan for using and sharing the technological products of this research, with appropriate timelines, will be assessed for its potential effectiveness and impact on other projects and the SCH overall. For NSF, reviewers will also be instructed to evaluate the proposed activities, their utility to the proposed research, and the SCH goals more broadly. Survivability: Credibility of the plan for continuing the development and transferring the artifacts and know-how after the expiration of research funding including steps toward commercialization and translation to the bedside. Impact: Potential for general impact on the development of techniques, environments or paradigms that will advance the provision of a system with patient-centered, evidence-based, prevention-focused and/or personalized care. Collaboration and Management: The extent to which the group is integrated, has a common focus and the quality of management and collaboration plans. Education and Training: The degree to which research and education are integrated and activities involve participation and training of students. Reviewers will assess the potential for involvement of motivated populations of young researchers in advancing health through science, technology, and behavior.

NSF Reviewer Proposal Rating

• Excellent Outstanding proposal in all respects; deserves highest priority for support.

• Very Good High quality proposal in nearly all respects; should be supported if at all possible.

• Good A quality proposal, worthy of support.• Fair Proposal lacking in one or more critical

aspects; key issues need to be addressed.• Poor Proposal has serious deficiencies.

14

Required Supplementary Documents• Data Management Plan (all)

– Definition of “data” and the adequacy of the management plan is determined by the community of interest through the process of peer review and program management

• Post Doctoral Mentoring Plan (only if postdocs involved)• Collaboration/Coordination Plan required for INT proposals• Human Subjects Protection (if necessary)• Vertebrate Animals (only if animal subjects involved)• Letters of Commitment

– time of participants not in the budget– facilities, data, and other resources– but no letters of recommendation or endorsement!

• List of Project Personnel

15

Please review and comment on these as well as the project description. If a required component is missing, tell the program officer immediately, as it may be cause for return without review.

Collaboration/Coordination Plan• Justifies the level of investment requested, and demonstrates

that the participating investigators will work synergistically to accomplish project objectives Required for all proposals.

• Typical elements:– specific roles of project participants in all organizations involved– how the project will be managed across all the investigators,

institutions, and/or disciplines– specific coordination mechanisms that will ensure cross-

investigator, cross-institution, and/or cross-discipline scientific integration (e.g., yearly workshops, graduate student exchange, project team meetings at conferences, use of the grid for videoconferences, software repositories, etc.)

– specific references to budget line items that support collaboration and coordination mechanism

16

Discussion Process• Lead reviewer

– may be same as scribe for some panels – starts with unbiased summary of proposal, for the benefit of

panelists who have not read it (≤ 3 minutes)– then gives own critical evaluation of proposal (2 minutes)

• Other reviewers add to or disagree with points made by lead

• Other panelists ask questions and comment based on own expertise and/or reading of proposal

• Scribe takes notes for NSF panel summary, asks questions if any required points have not been discussed

• Conflict of interest: Need to leave room during discussion and rankings

17

External Reviews

• If reviewer is not present at panel meeting• Review must be read by all panelists• An assigned panelist is the “voice” of the

external reviewer(s) during the discussion of the proposal

18

Panel Objectives• Provide informative feedback to the proposers

– Even the best proposals can benefit from advice– The weaker the proposal the more crucial the

feedback • Provide advice to NSF/NIH officers for making

award recommendations• Provide basis for decision making by higher-level

NSF/NIH management in approving (or not) funding or declination recommendations

19

The panel’s recommendations are advisory– funding decisions must also consider a variety of other issues.

Panel Outputs• Individual reviews for each proposal

– OK to modify reviews, including change of rating– Ensure your review text supports your rating– Be sure any modifications to reviews are recorded in FastLane!

These MUST be made BEFORE leaving your panel• NIH ratings & forms• NSF rating & panel summary for each proposal

– Initially framed by one reviewer who serves as scribe– Use the provided template!– Should reflect discussion (not just restate individual reviews)– Includes short, clear comments to help unsuccessful PIs improve

their proposals in the next competition– Should be written in 3rd-person and proof-read by all assigned

panelists

20

The proposer will receive...

Comments should be clear, informative, non-inflammatory, and non-discriminatory

Do not include identifying information about yourself, other panelists, or other proposals

21

verbatim copies of your NSF reviews

verbatim panelsummary

award or declination letter and context information

$

NATIONAL SCIENCE FOUNDATION4201 Wilson Boulevard

Arlington Virginia 22230

Dear Dr. Doe

The National Science Foundation hereby awards a grant of... NATIONAL SCIENCE FOUNDATION

4201 Wilson BoulevardArlington Virginia 22230

Dear Dr. Doe

I regret to inform you that the National Science Foundation is unable to support your proposal referenced above...

NSF Panel Recommendation Categories

• Highly Competitive – top priority for funding• Competitive – worthy of funding • Low Competitive – deficient on one or more

ways• Not Competitive – deficient in multiple ways,

do not encourage to revise• Not Discussed in Panel – triaged, major

deficiencies, or just not appropriate for this program

22Rank order only within HC and C

TriageThe panel may agree not to discuss proposals that received uniformly unenthusiastic reviews, < V scores. The triage decision will be based on unanimous consent by the panel. Any panelist may request that a proposal be discussed.• Triaged proposals are categorized as “Not

Discussed in Panel”• PIs receive verbatim copies of individual reviews• Triage Panel Summary.

23

In these cases individual reviews must be detailed and address all review criteria.

Presenter

Presentation Notes

If a proposal is discussed, it must be discussed completely, and a full Panel Summary must be produced.

Triaged Proposals

• Are categorized as “Not Discussed in Panel” for NSF and “Not Discussed” for NIH

• PIs receive verbatim copies of individual reviews• Triage Panel Summary

24

The Triage Panel Summary cannot be used if there has been any discussion of the proposal.

Presenter

Presentation Notes

If a proposal is discussed, it must be discussed completely, and a full Panel Summary must be produced. The only discussion permitted for a triaged proposal, is limited to: Asking whether anyone objects to triage for the proposal, i.e., whether anyone wants to discuss it. If no one wants to discuss it, there may be discussion over whether the reviews are adequately detailed to allow triage. If not, individuals may agree to revise reviews, and the triage question may be taken up again after the revision(s). Otherwise, the proposal must be discussed fully.

Review Process & Criteria

25

NSF proposals

NSF/NIH joint review

panelsreviews, ratings,

summaries, qualitative

recommendations

NSF award recommendations

NIH award recommendations

reviews, summaries, quantitative

scores

missions and priorities

Interagency funding meeting

ReviewersNSF PDNIH SRO

Proposals that NIH would like to fund will be selected and the PIs will be asked to

submit them (without any changes to the project description) in the NIH format

NIH Core Review Criteria

• Overall Impact = likelihood of a sustained, powerful influence on the research field(s)– Includes the significance of the project,

investigator qualities, innovation of the project, strength of the approach, research environment (including collaborations)

26

Presenter

Presentation Notes

Overall Impact. Reviewers will provide an overall impact/priority score and criterion scores to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following five core review criteria, and additional review criteria (as applicable for the project proposed). Significance. Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field? Investigator(s). Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project? Innovation. Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed? Approach. Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed? If the project involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed? Environment. Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements? Where applicable, the following items will also be considered: Protections for Human Subjects. For research that involves human subjects but does not involve one of the six categories of research that are exempt under 45 CFR Part 46, the committee will evaluate the justification for involvement of human subjects and the proposed protections from research risk relating to their participation according to the following five review criteria: 1) risk to subjects, 2) adequacy of protection against risks, 3) potential benefits to the subjects and others, 4) importance of the knowledge to be gained, and 5) data and safety monitoring for clinical trials. For research that involves human subjects and meets the criteria for one or more of the six categories of research that are exempt under 45 CFR Part 46, the committee will evaluate: 1) the justification for the exemption, 2) human subjects involvement and characteristics, and 3) sources of materials. For additional information on review of the Human Subjects section, please refer to the Human Subjects Protection and Inclusion Guidelines. Inclusion of Women, Minorities, and Children. When the proposed project involves clinical research, the committee will evaluate the proposed plans for inclusion of minorities and members of both genders, as well as the inclusion of children. Vertebrate Animals. The committee will evaluate the involvement of live vertebrate animals as part of the scientific assessment according to the following five points: 1) proposed use of the animals, and species, strains, ages, sex, and numbers to be used; 2) justifications for the use of animals and for the appropriateness of the species and numbers proposed; 3) adequacy of veterinary care; 4) procedures for limiting discomfort, distress, pain and injury to that which is unavoidable in the conduct of scientifically sound research including the use of analgesic, anesthetic, and tranquilizing drugs and/or comfortable restraining devices; and 5) methods of euthanasia and reason for selection if not consistent with the AVMA Guidelines on Euthanasia. . For additional information, see http://grants.nih.gov/grants/olaw/VASchecklist.pdf. Biohazards. Reviewers will assess whether materials or procedures proposed are potentially hazardous to research personnel and/or the environment, and if needed, determine whether adequate protection is proposed. Budget and Period of Support�Reviewers will consider whether the budget and the requested period of support are fully justified and reasonable in relation to the proposed research. For those proposals that are selected for funding consideration by participating NIH Institutes, the NIH will ask the applicant(s) to resubmit the proposal in an NIH-approved format directly to the Center for Scientific Review (CSR) at the NIH. Each of these NIH applications will be accompanied by a cover letter that associates the application with SCH. Applicants will not be allowed to increase the proposed budget or change the scientific content of the application in the resubmission to the NIH. These NIH applications, along with the summary statements generated based on the review, will be entered into the NIH IMPAC-II system.

NIH Additional Review Criteria• Protections for Human Subjects

– Risks, protections, potential benefits, importance– Data and safety monitoring for clinical trials

• Inclusion of Women, Minorities, and Children– Adequacy of the plans to include both genders, minorities and their

subgroups, and children (<18 years) as appropriate to the scientific goals of the research

• Vertebrate Animals• Biohazards

Not Scored• Adequacy of the Data Management Plan and resource sharing• Budget & Period of Support

27

Presenter

Presentation Notes

Overall Impact. Reviewers will provide an overall impact/priority score and criterion scores to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following five core review criteria, and additional review criteria (as applicable for the project proposed). Significance. Does the project address an important problem or a critical barrier to progress in the field? If the aims of the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field? Investigator(s). Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project? Innovation. Does the application challenge and seek to shift current research or clinical practice paradigms by utilizing novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the concepts, approaches or methodologies, instrumentation, or interventions novel to one field of research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed? Approach. Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project? Are potential problems, alternative strategies, and benchmarks for success presented? If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed? If the project involves clinical research, are the plans for 1) protection of human subjects from research risks, and 2) inclusion of minorities and members of both sexes/genders, as well as the inclusion of children, justified in terms of the scientific goals and research strategy proposed? Environment. Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements? Where applicable, the following items will also be considered: Protections for Human Subjects. For research that involves human subjects but does not involve one of the six categories of research that are exempt under 45 CFR Part 46, the committee will evaluate the justification for involvement of human subjects and the proposed protections from research risk relating to their participation according to the following five review criteria: 1) risk to subjects, 2) adequacy of protection against risks, 3) potential benefits to the subjects and others, 4) importance of the knowledge to be gained, and 5) data and safety monitoring for clinical trials. For research that involves human subjects and meets the criteria for one or more of the six categories of research that are exempt under 45 CFR Part 46, the committee will evaluate: 1) the justification for the exemption, 2) human subjects involvement and characteristics, and 3) sources of materials. For additional information on review of the Human Subjects section, please refer to the Human Subjects Protection and Inclusion Guidelines. Inclusion of Women, Minorities, and Children. When the proposed project involves clinical research, the committee will evaluate the proposed plans for inclusion of minorities and members of both genders, as well as the inclusion of children. Vertebrate Animals. The committee will evaluate the involvement of live vertebrate animals as part of the scientific assessment according to the following five points: 1) proposed use of the animals, and species, strains, ages, sex, and numbers to be used; 2) justifications for the use of animals and for the appropriateness of the species and numbers proposed; 3) adequacy of veterinary care; 4) procedures for limiting discomfort, distress, pain and injury to that which is unavoidable in the conduct of scientifically sound research including the use of analgesic, anesthetic, and tranquilizing drugs and/or comfortable restraining devices; and 5) methods of euthanasia and reason for selection if not consistent with the AVMA Guidelines on Euthanasia. . For additional information, see http://grants.nih.gov/grants/olaw/VASchecklist.pdf. Biohazards. Reviewers will assess whether materials or procedures proposed are potentially hazardous to research personnel and/or the environment, and if needed, determine whether adequate protection is proposed. Budget and Period of Support�Reviewers will consider whether the budget and the requested period of support are fully justified and reasonable in relation to the proposed research. For those proposals that are selected for funding consideration by participating NIH Institutes, the NIH will ask the applicant(s) to resubmit the proposal in an NIH-approved format directly to the Center for Scientific Review (CSR) at the NIH. Each of these NIH applications will be accompanied by a cover letter that associates the application with SCH. Applicants will not be allowed to increase the proposed budget or change the scientific content of the application in the resubmission to the NIH. These NIH applications, along with the summary statements generated based on the review, will be entered into the NIH IMPAC-II system.

Overall Impact

Score 1 2 3 4 5 6 7 8 9

High Medium Low

Evaluating Overall Impact

Evaluating Overall Impact: Consider the 5 criteria: significance, investigator, innovation, approach, environment (weighted based on reviewer’s judgment) and other score influences (e.g. human subjects)

5 is a good medium-impact application, and the entire scale (1-9) should always be considered.

e.g. Applications may be addressing a problem of highimportance in the field, but weaknesses in the criteria bring down the overall impact to medium.

e.g. Applications may be addressing a problem of moderateimportance in the field, with some or no technical weaknesses

e.g. Applications may be addressing a problem of moderate/highimportance in the field, but weaknesses in the criteria bring down the overall impact to low.

e.g. Applications may be addressing a problem of low or noimportance in the field, with some or no technical weaknesses.

e.g. Applications are addressing a problem of high importance/interest in the field. May have some or no technical weaknesses.

Overall Impact: The likelihood for a project to exert a sustained, powerfulinfluence on research field(s) involved

28

NSF-NIH Scoring GuideNIH

SCORENIH

PERCENTILE

1—2 10%

2—3 10 – 20%

3—4 20 – 30%

4—5 30 – 40%

5—6 40 – 50%

>6 >50% 0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

Perc

entil

e

Overall Impact Score

These are typical values, not rules.29

Panel Summaries• Use the template that is provided • Each panel summary must address:

– Intellectual Merit (strengths and weaknesses)– Broader Impacts (strengths and weaknesses)– Solicitation-specific criteria (strengths and weaknesses)– Panel recommendation and rationale– Suggested improvements (optional)

• Panel Recommendation– Recommendation category, and a sentence or two summarizing

reasons.– e.g., “The panel classified this proposal as Competitive. The principal

reasons are ...”This summary was read by all and the panel concurred that the summary

accurately reflects the panel discussion..

30

Presenter

Presentation Notes

The summary should reflect the principal points of discussion and the strengths and weaknesses that guided the panel’s recommendation. Scribe develops draft; other reviewers comment on substance and tone and suggest revisions as needed to develop a consensus summary that is approved by all.

Fit Summary to Recommendation• Highly Competitive

– Argue for funding priority.• Competitive

– Argue why it is worthy of funding– May have some concerns, advice to PI for

improvement if not funded• Low Competitive

– Point out aspects that are worthy,and aspects that are not

• Not Competitive– Explain what it wrong and why it is not a good idea for

research or is not a good fit for the program• Not Discussed in Panel

– Must use Triage Panel Summary

31

Mainly to convince NSF management.

Mainly for the benefit of the PI.

Presenter

Presentation Notes

The summary should reflect the principal points of discussion and the strengths and weaknesses that guided the panel’s recommendation. Scribe develops draft; other reviewers comment on substance and tone and suggest revisions as needed to develop a consensus summary that is approved by all.

Minimize Bias in Evaluation

• Be conscious of possible sources of bias• Rate only on explicit review criteria

– Identify specific strengths and weaknesses • Point to specific evidence in supporting

judgments• Avoid bias “creep”

– Small bias in same direction has large effect over time– Very small differences in treatment can have major

consequences in salary, promotion and prestige32

Evaluation & Writing Guidelines

NSF and NIH thank you!

33

Fastlane: Panelist Functions

34

Using the Interactive Panel System• Log into FastLane• Be sure to not to un-check the

box to DISABLE the refresh!• Go to “Meeting Sign-In” before

“Interactive Panel System”

35

x

Meeting Sign-In

36



Presenter

Presentation Notes


Interactive Panel System

37



Presenter

Presentation Notes


Scribe’s window for Panel Summary

38

– S

Pasting Summary Into Form• Scribes - prepare your draft of the panel summary in MS

Word (Notepad) and then copy-and-paste it into text box under “summary” tab

• Beware that Fastlane will convert many special symbols to question marks “?”. (See more detailed instructions to come.)

• Click “Save” and then click “Submit Draft for Comment.”

39

Panelist Comment Form• All panelists must read the panel summary,

and can suggest corrections or changes by using the “Write Comment” tab.

40

Viewing Comments– All comments made can be viewed in the “View

Comment” tab– Scribes, be sure to watch for comments from

reviewers and program officer!– When the panel summary looks good, reviewing

panelists must approve the panel summary by commenting that it “looks good”, “OK”, “I approve”, etc.

41

Submit for Final Approval• When all relevant panelists are comfortable with

the text and have approved, the moderator will let the scribe know the panel summary can be submitted for approval. The scribe will then click the “Submit for Approval” button.

42

Revising a Review or Rating

43

Do this in a new browser session, not while in the Interactive Panel system!

Presenter

Presentation Notes


National Science Foundation -...

Documents

Transcript of National Science Foundation -...