Future and Emerging Technologies Features and evaluation ... · Future and Emerging Technologies...

1 of 51

Future and Emerging Technologies

Features and evaluation process

Briefing for Remote Evaluators 08 November 2017

Antonio LOREDAN Call Coordinator FET-Open Unit

2 of 51

2

You can send your questions to the following address:

[email protected]

they will be answered at the end of the presentation

mailto:[email protected]











3 of 51

1. FETOPEN RIA - Main Features

2. Confidentiality and Conflict of Interest

3. Admissibility and Eligibility Check

4. Evaluation Process

5. Remote phase: IERs, REs and QCs roles

Agenda:

4 of 51






5 of 51

Excellent Science in Horizon2020

INDUSTRIAL LEADERSHIP € 16,5 billion

SOCIETAL CHALLANGES € 28,6 billion

EXCELLENT SCIENCE

€ 24,2 billion

OTHER * € 6 billion

H2020 budget € 74,8 billion

*OTHER: • Spreading excellence & widening participation • Science with and for society • JRC • EIT

ERC 54%

FET 11%

MSCA 25%

Research infrastructure

s 10%

6 of 51

FET-Open Research and Innovation Actions: supporting early-stages of research to establish a new technological possibility.

Collaborative projects up to € 3 Mio funding (indicative)

Single step submission, '1+15' pages

No thematic restriction, no emphasis on any subject

Bottom-up, but targeted - not blue sky research

Typical FET project result is a proof of a concept

Scope defined by FET gatekeepers

7 of 51

SCOPE: […] Proposals are sought for collaborative research with all of the following characteristics ('FET gatekeepers'):

Gatekeepers

Long-term vision Interdisciplinary

Novelty

Foundational

High-risk

S&T breakthrough

Far beyond… Ambitious

Not incremental

Open & Agile practices

New solutions & synergies

8 of 51

Long-term vision new and radical long-term vision of a science- and

technology-enabled future that is far beyond the state of the art

9 of 51

Breakthrough S&T targeted

Scientifically ambitious and technologically concrete breakthroughs plausibly attainable within the life-time of the project Ambitious

10 of 51

Foundational

Far beyond…

The breakthroughs must be foundational in the sense that they can establish a basis for a new line of technology not currently anticipated

11 of 51

Novelty

Not incremental

New Ideas and concepts, rather than the application or incremental refinement of existing ones

12 of 51

High-risk

Open & Agile practices

Balancing the high risk versus being utterly unrealistic . High-risk is not a synonym with not-doable

13 of 51

Interdisciplinary

New solutions & synergies

The proposed collaborations must go beyond current mainstream collaboration configurations in joint S&T research, and must aim to advance different scientific and technological disciplines together and in synergy towards a breakthrough

14 of 51

FET-Open Research and Innovation Actions

Expected impact

Establish baseline of feasibility and innovation potential

Strengthening European leadership in the early exploration of

visionary, new and emerging technologies

New R&I practices

15 of 51






16 of 51

The expert must keep his/her work under the Contract strictly confidential, and in particular:

a) not disclose (directly or indirectly) any confidential information relating to proposals or applicants, without prior written approval by the Agency.

b) not discuss proposal(s) with others (including other experts or Agency staff that are not directly involved in the evaluation of the proposal(s), except during evaluation meetings and with prior approval by the responsible Agency staff.

c) not disclose:

- details on the evaluation process or its outcome, without prior written approval by the Agency

- details on his/her position/advice;

- the names of other experts participating in the evaluation.

Social media: Take great care not to post pictures or comments on evaluation matters through social media

17 of 51

The expert must keep his/her work under the Contract strictly confidential, and in particular:

d) not communicate with applicants (including linked third parties or other third parties involved in the actions) during the evaluation or afterwards — except in panel hearings.

e) may not remove from the premises any documents, material or information on the proposal(s) or on the evaluation.

f) is responsible for ensuring adequate protection of electronic documents and information and for returning, erasing or destroying all confidential information after the end of the evaluation.

Social media: Take great care not to post pictures or comments on evaluation matters through social media

18 of 51

An expert has a conflict of interest if : was involved in the preparation of the proposal

would benefit directly or indirectly if the proposal is accepted

has a close family (or other personal) relationship with any person representing an applicant legal entity

is a director, trustee or partner or is in any way involved in the management of an applicant legal entity

is employed or contracted by one of the applicants or a named subcontractor

is a member of an ‘advisory group’ (set up by the Commission to advise on the preparation of EU or Euratom Horizon 2020 work programmes) in an area related to the call

is a ‘national contact point (NCP)’ or is directly working for the ‘Enterprise Europe Network (EEN)’

is a member of the H2020 Programme Committee

19 of 51

The Agency decides on whether there is a conflict of interest (on the basis of circumstances, available information and related risks), if the expert: was employed by one of the applicants in the last three years

is involved in a contract, grant agreement, grant decision, management structure (e.g. member of management or advisory board etc.) or research collaboration with an applicant/fellow (or had been so in the last three years)

is in any other situation that could cast doubt on their ability to participate in the evaluation of the proposal impartially (or that could reasonably appear to do so in the eyes of an external third party).

20 of 51

You must inform the Agency as soon as you become aware of a COI

If there is a COI for a certain proposal you

cannot evaluate the give proposal

might be excluded from the panel review for the given proposal

The Agency will determine if there is a COI on a case-by-case basis and decide the course of action to follow

If you knowingly hide a COI, you will be excluded from the entire evaluation and your work declared null and void

The allowance/expenses you claimed may be reduced, rejected or recovered

Your contract may be terminated

21 of 51

If a conflict becomes apparent at any stage of the evaluation, you must immediately inform us.

If a conflict is confirmed, you must stop evaluating the proposal concerned. Any comments and scores already given by you will be disregarded. If necessary, you will be replaced.

22 of 51






23 of 51

ADMISSIBILITY

To be considered admissible, a proposal must:

1. Be readable, accessible and printable.

2. Be complete

3. Include a draft plan for the exploitation and dissemination.

24 of 51

A proposal/application will only be considered eligible if: (a) Its content corresponds, wholly or in part, to the topic description for

which it is submitted

(b) At least three legal entities are present in the consortium. And :

Each of the three must be established in a different EU Member State or Horizon 2020 associated country.

All three legal entities must be independent of each other.

ELIGIBILITY

25 of 51

Part A: Administrative part of the proposal

Part B: Technical Annex : Section 1-3 (16 pages maximum)

Cover page

Section 1: S&T Excellence

Section 2: Impact

Section 3: Implementation

Section 4-5

Section 4: Members of the consortium

E.g. legal entity, CV, subcontract, third party

Section 5: Ethics and Security

Ethics self-assessment & supporting documents

Security checklist

Section 4 & 5 are not

covered by the page

limit.

Section 1,2 & 3 are strictly

limited to 15 pages!

WATERMARK!!!

Cover page strictly limited to 1 page!

Completeness and page limits

26 of 51

Admissibility and eligibility of proposals have been checked by the Agency - however, if you spot an issue relating to eligibility, please contact us.

27 of 51

27


[email protected]













28 of 51






29 of 51

FET-Open RIA in a nutshell

30 of 51

FET-Open RIA in a nutshell

Calls Number of

eligible proposals

Number of grants

Total Requested

(M€) Success rate

Sep-14 639 24 78,1 3,70%

Mar-15 665 11 41 1,65%

Sep-15 800 11 37,8 1,37%

May-16 544 23 87,8 4,22%

Jan-17 365 26 84,8 7,12%

Total 3019 95 329,5

Sep-17 395 26* 84*

*Estimated

31 of 51

Call Schedule

32 of 51

Feedback in 5 months

Ethics screening/

assessment

Panel review Final comment & score

Cross-reading By panel membres

Quality check of

IER with support from Vice Chairs

Remote evaluators write

Individual Evaluation Report

Eligibility check

REA

Proposal submission

Collated report Check by remote

experts

Expert Assignment

Collate IER comments, median score calculated

Conflict of interest check at any stage of evaluation

Iterations if necessary

Each proposal is allocated to 4 remote evaluators

Each evaluator drafts comments and assigns scores

Draft ESR created

Evaluation process (1)

New in Jan 17 cut off

33 of 51


Ethics screening/

assessment



Quality check of




Eligibility check

REA

Proposal submission


experts

Expert Assignment


Each evaluators drafts comments and assigns scores


• Systematic cross reading of proposals

• Panel members briefed to carefully look at 'diverging' opinions

34 of 51


Ethics screening/

assessment



Quality check of




Eligibility check

REA

Proposal submission


experts

Expert Assignment




• Discussion of all 'highly scored' proposals

• Low scored proposals are discussed too

• Special attention to 'diverging' opinions

• Final score decision by consensus or vote if necessary

• Panel should add comments on divergences

35 of 51


Ethics screening/

assessment

Panel review Final comments &

score


Quality check of




Eligibility check

REA

Proposal submission


experts

Expert Assignment




Evaluation Summary Report (ESR): collation of all (4) evaluators' comments, per sub-criterion, which may be mutually contradicting - full transparency

36 of 51

Main Actors: • Remote Evaluators: ~600

o Write remotely good quality individual evaluation reports and assign corresponding scores

o Submit individual evaluation reports within deadline

• Quality Controllers (QC): ~70 o Fine tuning of proposals' descriptors o Check the quality of individual evaluation reports

• Cross Readers (CR): ~70 o Cross-read proposals o Take part in the panel review in Brussels

• In addition - Independent Observer

37 of 51

Observer

Appointed by the Agency may monitor the remote evaluation, to ensure a high quality evaluation

He checks the functioning and running of the overall process

He advises, in his report, on the conduct and fairness of the evaluation sessions and, if necessary, suggests possible improvements

He does not evaluate proposals and, therefore, does not express any opinion on their quality

38 of 51






39 of 51

Remote evaluation and quality control (8 November– 7 December 2017)

Remote phase

40 of 51


[email protected]













41 of 51

Remote phase and your role (1):

Your role is to draft Individual Evaluation Reports (IERs) with your assessment of the proposals in accordance with the evaluation criteria.

Disregard excess pages marked with a watermark

Check to what degree the proposal is relevant to the call or topic

42 of 51

• Read the proposal and evaluate it against the evaluation criteria

Without discussing it with anybody else

As submitted - not on its potential if certain changes were to be made

Do not penalise applicants that did not provide detailed breakdown costs – they are not required

• Complete an Individual Evaluation Report (IER)

Give your view on operational capacity

Give comments and scores for all evaluation criteria (scores must match comments)

Explain shortcomings, but do not make recommendations (e.g. no additional partners, work packages, resource cuts)

• Sign and submit the form in the electronic system

Look at the substance: Some proposals might be

handicapped by language difficulties, others

deceptively well written

Remote phase and your role (2):

http://ec.europa.eu/research/participants/docs/h2020-funding-guide/grants/applying-for-funding/submit-proposals_en.htm

43 of 51

The quality of the IER is paramount

There will be no consensus meetings! No grant negotiation phase

The IERs remains unchanged at the panel stage and are then sent to applicants, with some additional comments if necessary.

The aim of the IER is to give:

A clear assessment of the proposal based on its merit

Clear feedback on the proposal’s shortcomings/weaknesses and strengths.

Individual Evaluation Report (IER)

44 of 51

Remote evaluators (RE) tasks:

RE evaluates each proposal as submitted, not on its potential

If RE shortcomings (other than minor ones and obvious clerical

errors), reflect those in a lower score for the relevant criterion

RE explains the shortcomings, but cannot make recommendations

Proposals with significant weaknesses that prevent the project

from achieving its objectives must not receive above-threshold

scores

45 of 51

Interpretation of the scores

The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.

Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.

Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.

Good. The proposal addresses the criterion well, but a number of shortcomings are present.

Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.

Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.

0

1

2

3

4

5

46 of 51

As remote evaluator you are requested to:

'Accept to evaluate' each proposal asap but not later than within 2 days!

Submit at least 2 (or all if less than 2 ) proposals assigned to by November 15

Submit at least 4 (or all if less than 4) proposals assigned to you by November 20

47 of 51

IER will be quality checked by a Vice Chair Quality Controller (QC) for:

Relevance and completeness of the comments

Consistency between comments and scores

Inappropriate language

Immediately after an evaluator submits the IER the quality control starts

Quality check of IERs (I)

48 of 51

Quality check of IERs (II)

QC provides feedback to REA in case an IER needs to be re-opened.

If applicable, you should revise a given IER based on REA request, following input from QC.

QC verifies that you have identified strengths and weaknesses of the proposal under the 3 criteria.

Your comments have to be relevant within each sub-criterion.

49 of 51

Feedback to proposers – Evaluation Summary Report (ESR)

Collation of all evaluators' comments , per sub-criterion, which may be

mutually contradicting (no consensus) - full transparency

Proposal score calculation (per criterion) - median of the scores from

individual evaluators

Final score per criteria is decided by the final panel review

Total final score for the proposal is calculated as the weighted sum of the

final scores from the 3 evaluation criteria

Final panel review adds also some additional comments

50 of 51

Remote Evaluators FET-Open RIA, Cut-off date 27 September 2017; Useful Info

https://ec.europa.eu/info/remote-evaluators-fet-open-ria-cut-date-27-

september-2017-useful-info_en

https://ec.europa.eu/info/remote-evaluators-fet-open-ria-cut-date-27-september-2017-useful-info_en























51 of 51

Thank you for your attention!

Future and Emerging Technologies Features and evaluation ... · Future and Emerging Technologies...

Documents

Transcript of Future and Emerging Technologies Features and evaluation ... · Future and Emerging Technologies...