Monitoring and assessing research performance€¦ · Dr. Walid HASSAN Research Consultancy...

November 2017

Monitoring and assessing research performance

November 2017

On ResearchBenchmarking and Valuation (Translation)

Dr. Walid HASSANResearch Consultancy Services Manager

The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man”..George Bernard Shaw, Man and Superman (1903) "Maxims for Revolutionists"

4

CLARIVATE ANALYTICS: A GLOBAL FOOTPRINT

1. With more than 4,200 staff worldwide, Clarivatesupports customers through curated editorialcontent, leading professional services, customproduct and service development and muchmore.

2. Over 7,000 leading government and academicinstitutions, and research intensive corporationsuse the Web of Science for its 100 years worth ofresearch.

3. Headquartered in the US, we are a global,information-led company operating in over 100countries around the world.

5

BIG DATA CAPABILITIES

THOMSON INNOVATION

• DWPI – world’s most comprehensive patent data set

• 4.5 Release and Smart Search – find what your are looking for quickly, easily

• ThemeScape – easily visualize how competitor portfolios overlap

6

OUR PEOPLE: CONNECTING ACROSS THE GLOBE

7

Content

� Top 2016 Discoveries

� Situation Analysis- Brazil

� Back to basics, Challenges

� On R&D Strategies

� Benchmarking

� Case Studies

8

Top 2016 DiscoveriesSlight Genetic Mutation 800 Million Years Ago Leads To Multicellular Life

Researchers found that an ancientmolecule, GK-PID, is the reason single-celled organisms started to evolve intomulticellular organisms approximately 800million years ago . The molecule hasbeen found to be like a molecularcarabiner able to pull chromosomestogether to latch them onto the inner wallof a cell membrane when division occurs.This allows cells to copy properly andavoid becoming cancerous.

Photo credit: TenOfAllTrades /Wikimedia

9

A Ninth Planet Was Discovered In The Solar System

California Institute of Technologypresented evidence that a ninth planettruly does exist with an orbital period of15,000 years.The astronomers who published theirdiscovery have calculated that there is“only a 0.007 percent chance, or aboutone in 15,000, that the clustering could bea coincidence.” Presently, Planet Nineremains hypothetical, but astronomershave calculated its orbit to be quitemassive. If it does exist, the planet wouldlikely be approximately 2–15 times themass of Earth and orbit between 200 and1,600 Astronomical Units (AU) from theSun. An AU is 150,000,000 kilometers,which means that the planet could orbit asfar from the Sun as 240,000,000,000kilometers.

Photo credit: Wikimedia

Top 2016 Discoveries

10

An Eternal 5D optical data Storage

University of Southampton: Scientists havesuccessfully used nano-structured glass to create aprocess for recording and retrieving data. The storagedevice is a small glass disk about the size of anAmerican quarter that can hold 360TB of data andremain intact up to 1,000°C. This means that itsaverage shelf life when held at room temperaturewould be approximately 13.8 billion years (Roughlythe same amount of time the universe has existed).

Photo credit: southampton.ac.uk


11

Stem Cells Injected Into Stroke Patients Re-Enable Patient To Walk

A clinical trial held at StanfordUniversity School of Medicineinjected modified human stem cellsdirectly into the brains of severalchronic stroke patients. Theprocedures were all successful withno negative effects described fromthe injection and only mildheadaches as a result of theprocedure, which was performedon mildly anesthetized patients. All18 showed significant healing longafter any healing is expectedfollowing a stroke (a period of sixmonths)Photo credit: Nissim Benvenisty


12

Dr. Walid Hassan

11/13/2017

Private Company SpaceX Successfully Lands A Rocket Vertically


13Where We Are

14How We Look

15Who?

16How? (% of Documents Cited)

17How? (% of Hot Papers)

19Who is concerned?

Research Process

PhD Students/Res

earchers

Research Managers`

Decision Makers

� Researchers: PhD Students and Researchers� Research Managers: Research unit managers, department managers, etc.. � Decision makers: Research Laboratory Managers, Institution Management, etc..

20Challenges and Repartition

The Start Point

Researchers are challenged at the first moment, when defining their research subject area

22

Topics Strategy

35 USD billions has been wasted developingthings that are already documented in a patentspecification .(Source: British Patent Office,2000)

23Patents Versus Publications

Publications

Patents

R&D STRATEGY

25

—TYPES OF RESEARCH

26

—PATHWAY FROM RESEARCH TO IP, STRATEGY

Fundamental

Publications

Citations, Visibility

Research Community (Academia )

Applied

Patents

Licensing

Industry

Theories, methodologies

Applications

IPKnow HowV

alua

tion

27

Positioning of needs: University

28

University: Opportunities and Challenges

PUBLISH, READ AND CITE

30

Publish or Perish

Publish ………. or Perish

They Say:

Something is Missing !!!!!!!!

31

Publishing: Problem Positioning

32

Today: What to Read

33

Contradictions?

Need a Benchmark

Research

papers?

Need a Model

34

EUGENE GARFIELD’S “ASSOCIATION OF IDEAS INDEX”

A benchmark..

A Model

35

Challenges

Why to publish?

Where ?

When ?

How?

With Whom?

36

Why we Publish

Diffusion of Knowledge

Promotion

Sharing Ideas

Community belonging

Research Valorization

Getting Feedback

Professional Profile

37

Where to Publish

• Which journal? Indexed?

• How journals are classified? Which is better?

• Should I look at the Impact factor?

• Would my publications be accepted in high-IF journals?

• Open Access?

BENCHMARKING

The main Pillar of Strategic Planning

39

BENCHMARKINGBENCHMARKING

Strategizing the Step from A to B

40BENCHMARKING IN RESEARCH

41GOALS AND OBJECTIVESIn

pu

ts Human Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research

Education & Training

Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact

International Reputation

Increased Funding

42KPI’s

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

Where is our

funding coming

from?

How much

funding do we

receive from

various

sources?

Who are the Who are the leaders in interested

fields?

What connections can we take

advantage of to recruit top

talent?

How much

research do we

disseminate?

What

publications

channels do we

use most

often?

How do we

compare to

similar

institutions?

How are we

perceived by

the research

community?

43PLANNING

ESTOU FAZENDO ÀS COISAS CERTAS?

• Theory Evaluations

• Mission statement development

• Needs assessments

• Strategic planning

ESTOU FAZENDO ÀS COISAS CERTAS CERTO?

• Process Evaluations

• Regular progress reports

• Monitoring databases and user interface systems

• Resource usage tracking

ESTOU FAZENDO O SUFICIENTE DAS COISAS CERTAS PARA FAZER A DIFERENÇA?

• Outcome Evaluations

• Output tracking

• Performance benchmarking

• Longitudinal attribution

44

Who are the stakeholders?

Executives

Researchers

Funders

Partners

Public

What domains are being addressed?

Need

Design

Implementation

Impact and outcome

Efficiency

How is the information presented?

Summary

Detailed report

Presentation

Infographic

Campaign

What are the expected returns?

Awareness

Advocacy

Accountability

Action

Assets

CAPITALISE

45

CAPES

46Where We Are

47Organizations

48

Não é possível exibir esta imagem no momento.

Organizations (Documents in Top 1%)

Dr. Eng. Walid HassanResearch Consultancy Services Manager

Email: [email protected]

Twitter: @Walid_Hassan_TR

Linked In: Walid Hassan, PhD

50

Case Study: University X (Optional)

Overview of a Brazilian University’s

research performance.

52

Evaluation of

University X’s

Research Activity,

2000-2016

Use case: exploring how bibliometric data can indicate institutional performance

across multiple components of a university research activity framework.

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

53

Evaluation of

University X’s

Research Activity

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

54

Funding

55

Acknowledged

Funding

(2008-2016)

56

Acknowledged

Funding

(2008-2016)

CNPq and CAPES excluded

57

Evaluation of

University X’s

Research Activity

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

58

Partnerships

59

Evaluation of

University X’s

Research Activity

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

60

Publication output

by year

Total (2000-2016): 142,077 publications

61

Publication output

by CAPES area

6 - Health Sciences, 41.0%

3 - Biological Sciences, 27.2%

5 - Exact and Earth Sciences, 25.8%

4 - Engineering, 12.2%

1 - Agricultural Sciences, 9.1%

9 - Multidisciplinary, 9.1%

2 - Applied Social Sciences, 3.7%

7 - Humanities and Social Sciences, 2.4%

8 - Linguistics, Literature and Arts, 0.2%

Note: Does not sum to 100% due to

publications in multiple categories

62

Evaluation of

University X’s

Research Activity

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

63

Collaborations

64

Brazilian

Collaborations

65

International

Collaborations

66

Evaluation of

University X’s

Research Activity

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

67

Impact as

Citation counts

68

Times Cited

and

Document count

69

Times Cited

and

Normalized Citation

Impact

70

Scientific ImpactUsing normalized citation measures

to indicate the impact of inputs, activities, and outputs

Example: Category Normalized Citation Impact

Where:

e = the expected citation rate or baseline,

c = Times Cited,

p = the number of papers,

f = the field or subject area,

t = year,

d = document type,

n = the number of subjects a paper is assigned to and

i = the entity being evaluated (institution, country, person, etc).

The Category Normalized Citation Impact (CNCI) of a document is calculated by dividing the actual

count of citing items by the expected citation rate for documents with the same document type, year

of publication and subject area.

CNCI

0,5

0,6

0,7

0,8

0,9

1

1,1

World Avg, 1.0

University X, 0.88

71

Acknowledged

Funding

(2008-2016)

CNPq and CAPES excluded

US DOE

US NIH US NSF

72

Acknowledged

Funding

(2008-2016)

CAPESNCPq

73

Output and Impact

by CAPES category6 – Health Sciences

5 – Exact and Earth Sciences

7 – Humanities and Social Sciences

74

Sub-analysis of

CAPES category

“Exact and Earth

Science”

75

Sub-analysis of

CAPES category

“Humanities

and

Social Sciences”

76

Collaborations,

Country-level

USA

Netherlands

77

Percentage of

papers with

international

collaborations by

year

Total percentage

2000-2016: 29.72%

78

Journal analysis

using citation

impact normalized

to the journal

79

Evaluation of

University X’s

Research Activity

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

80

Evaluation of

University X’s

Research Activity

Inp

uts Human

Resources

Funding

Facilities

Administration

Social Capital

Act

ivit

ies Research


Partnership

Recruitment

Thought Leadership

Ou

tpu

ts Publications

Graduates

Collaborations

Innovation

New Resources

Ou

tco

me

s Scientific Impact

Societal Impact


Increased Funding

81

Case Study: The Impact Factor and it’s

Usage

(Optional)

82

The Impact FactorThe impact factor is a measure of the frequency with which the average

article in a journal has been cited in a particular year. The JCR also lists

journals and their impact factors and ranking in the context of their

specific field(s).

83

Citations and IF variations

Dr. Walid Hassan

11/13/2017

84

Categories/Quartiles

- J1 IF1

- J2 IF2

- J3 IF3

- J4 IF4

- J5 IF5

- J6 IF6

- J7 IF7

- ------------

25%- Q1

25%- Q2

25%- Q3

25%- Q4

Q1

Q1

85

Case Study: Building a PhD Subject

(Optional)

86

—SELECTING THE RIGHT RESEARCH TOPIC

Example: Image Reconstruction

87

—TREND (NUMBER OF DOCUMENTS PER YEAR IN WOS)

88

—NORMALIZED CITATION IMPACT

89

—PATENT ANALYSIS: IMAGE RECONSTRUCTION

90

—THEMESCAPE (DERWENT INNOVATION)

Red Dots: Alive Patents

91

WEB OF SCIENCE (WOS) DATABASES

SCIENCE CITATION INDEX EXPANDED

•41.2 million records

•1900-present

SOCIAL SCIENCES CITATION INDEX


•1900-present

ARTS & HUMANITIES CITATION INDEX


•1975-present

CONFERENCE PROCEEDINGS CITATION INDEX


•1990-present

BOOK CITATION INDEX

•977,199 records

•2005-present

CURRENT CHEMICAL REACTIONS

•212,622 records

•1985-present

INDEX CHEMICUS

•337,305 records

•1993-present

DATA CITATION INDEX


•1900-present

CURRENT CONTENTS CONNECT


•1998-present

BIOSIS CITATION INDEX


•1926-present

ZOOLOGICAL RECORD

•4 million records

•1864-present

MEDLINE

•22 million records

•1950-present

DERWENT INNOVATION INDEX

•22.9 million basic inventions

•1963-present

A comprehensive gateway to the most relevant research information: reliable, integrated, multidisciplinary

Monitoring and assessing research performance€¦ · Dr. Walid HASSAN Research Consultancy...

Documents

Transcript of Monitoring and assessing research performance€¦ · Dr. Walid HASSAN Research Consultancy...