Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact John Barnett Scholarly...

45
Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact John Barnett Scholarly Communications Librarian Jennifer Chan Assistant Scholarly Communications Librarian Office of Scholarly Communication and Publishing University Library System University of Pittsburgh

Transcript of Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact John Barnett Scholarly...

Here Comes the Sunburst:Measuring and Visualizing Scholarly Impact

John Barnett

Scholarly Communications Librarian

Jennifer Chan

Assistant Scholarly Communications Librarian

Office of Scholarly Communication and PublishingUniversity Library System

University of Pittsburgh

Here Comes the Sunburst:Measuring and Visualizing Scholarly Impact

University of Pittsburgh

Pittsburgh campus + regional campuses in Bradford, Greensburg, Johnstown, and Titusville

16 undergraduate, graduate, and professional schools

456+ degree programs 2012: conferred 8,949 degrees

University of Pittsburgh

Top 10 American higher ed. in federal funding (NSF)

Top 5 in annual research support (NIH)

5,369 faculty; 4,470 full-time faculty Research conducted: more than 300

centers, institutes, laboratories, clinics

University Library System

ARL

22nd largest academic library system in North America

25 libraries; 6.6 million volumes

279,000 current serials

Sum of the Parts

Liaison Librarians

ULS Department of

Information Technology

Office of Scholarly

Communication and Publishing

Why Pitt?

Strategic goal: Innovation in scholarly communication

Providing services that scholars understand, need, and value

Putting ourselves in faculty “spaces”

Re-envisioning our librarian liaison program

Deepening our understanding of scholarly communications issues

Why PlumX? Making research “more assessable

and accessible”– Gathering information in one place– Making it intelligible and useful

Measuring and visualizing research impact

Correlating metrics from traditional and new forms of scholarly communication

Allowing researchers, labs, departments, institutions to track real-time scholarly impact

Promoting research, comparing with peers, connecting with new research

Altmetrics Project Timeline

Spring 2012: • First meeting with

Plum Analytics

Fall 2012• Gathered data

from pilot participants

Spring 2013• Faculty surveyed;

enhancements made

Pilot project aims

Develop a tool for measuring and visualizing research impact

Gathering information in one place

Intelligible and useful

Impact in social media and other scholarly communication methods

Traditional measures counted as well

See where to disseminate works to increase impact

Traditional vs. new

•Traditional measures are also counted

•Findings are complementary to conventional methods of measuring research impact (e.g., H-Index)

•Not intended to replace them

New measures

More comprehensive: Altmetrics = ALL METRICS– Citations– Usage– Captures– Mentions– Social Media

Covers impact of online behavior– Because scholars increasingly work online

Measures impact immediately– Because citation counts take years to appear in literature

Pilot Process

Created Altmetrics Task Force

Engaged Liaison Librarians

CV receipt and records creation

Pilot Project Participants

• 32 researchers, various disciplines

• 9 schools

• 18 departments

• 1 complete research group

• Others joined as they learned about the project

Pilot Project Participantsdiscipline

online behavior

level of career advancement

Selected faculty

participants, diversified by:

Technologies

Internal– IR built on Eprints Platform– Sharepoint– Microsoft Office Suite– PMID/DOI data import tool

External– PlumX– DOIs– PMID

Data collection for pilot project

• Created records in D-Scholarship@Pitt, our institutional repository

• Focused on articles, books, book chapters, proceedings

• Scholarly output with standard identifiers• DOI, ISBN, PubMed

ID, official URL, etc.

• Scholarship produced since 2000

Other Library work

• Developed guidelines to standardize record creation

• Data entry from faculty CVs into IR (2 to 3 student workers with QA by librarians)

• Librarian liaisons and other staff trained in record creation

• SharePoint site used to track work completed

• Coordination with pilot faculty

• Gathered feedback and administered online survey

Sharepoint

Altmetrics Meetings Minutes

Faculty CVs

Excel spreadsheets

Word docs

External Data Sources

Metadata sources

Faculty CVs . . . But verify metadata!

Books: PittCat, WorldCat, Books in Print, publisher sites, online retailers

Journals: Serials Solutions list, journal websites, JournalSeek, UlrichsWeb, DOAJ, PubMed

Conference presentations: Websites, PittCat, indexes, WorldCat

PMID Import Tool

Custom build by SysAdmin for Eprints Platform

Utilizing PMIDs from PubMed, able to import records that prepopulate metadata fields– Item Type, Title, Abstract, Creators, Publication Title, ISSN,

Volume/Issue, Page ranges, Date and Date type, DOI, MeSH Headings, Grant Information, Keywords, etc.

Data Ingestion

IR

External Data

Ingestion

CV input

Self-archiving

Full-text sources

DOAJ

ERIC

PLOS

SSRN*

Other repositories*

Federal government websites*

Conference websites*

* Use with caution

Plum Analytics processing activities

Harvest records from Pitt IR for each participant

Harvest additional online artifacts NOT in Pitt IR

Use data mining to harvest publically available metrics from hundreds of sites on the Web

C

r

e

a

t

e

v

i

s

u

a

li

z

a

ti

o

n

s

t

o

d

i

s

p

l

a

y

m

e

tri

c

s

o

n

P

l

u

m

X

i

n

t

e

rf

a

c

e

Key features

Faculty profiles

Online ‘artifacts’– Article– Book– Book chapter– Video– Etc.

Impact graph

Sunburst

Faculty profile

Online ‘artifact’ display

Impact graph

Sunburst

Feedback

• Solicited via email and online survey

• Generally positive in most cases

• Data corrections• Errors in profiles• Links to wrong data• Quickly corrected by Plum staff

• Requests for results from additional online sources (Google Scholar, SlideShare, Reddit, etc.)

• PlumX collects data from these but did not gather information in advance for profiles

The survey says

Surveyed pilot project faculty in spring 2013

@ 1/3rd responded to the survey

Meaning 13 out of 32 participants responded

Accurate and useful data

Summary profile data page0

10

20

30

40

50

60

70

80

90

15.38

7.69

76.92

0

Strongly disagreeDisagreeAgreeStrongly agree

The bar graph

Usefulness of interactive bar graph0

10

20

30

40

50

60

70

25

58.33

16.67

Strongly disagreeDisagreeAgreeStrongly agree

0

The sunburst

Usefulness of sunburst0

10

20

30

40

50

60

41.67

50

8.33

Strongly disagreeDisagreeAgreeStrongly agree

0

Traditional & new measures

Conveying traditional and new measures0

10

20

30

40

50

60

36.36

54.55

9.09

Strongly disagreeDisagreeAgreeStrongly agree

0

Usefulness of altmetrics

The value of altmetrics0

10

20

30

40

50

60

18.18 18.18

54.55

9.09

Strongly disagreeDisagreeAgreeStrongly agree

Learning something new

Learning something new about my own research0

5

10

15

20

25

30

35

40

9.09

36.36

27.27 27.27

Series 1Series 2Series 3Series 4

Comments

Affiliations/bio inaccurate or has missing information

“Mentions” by whom & when?

Publications misclassified– Books vs. conference proceedings

Data not collected– Google Scholar– Slideshare

Comments

Filter out unwanted information

Data are wrong—and not useful

Overabundance of information in sunburst

“I only care what a select group of scholars thinks of my work”

“I did not find this useful for my discipline”

Observations

Lacked information about faculty practices

Are the results useful to all faculty, all disciplines?

May appeal more to faculty who are early in their careers or whose work is more contemporary

Will the data be used against faculty or programs?

Labor-intensive strategy

When it comes down to it . . . Does anyone care?

Embeddable widgets(in development)

For researchers, to add to:• their own Web pages• department directories• IR researcher profile page

For individual artifacts,to build article level metrics for imbedding in:

• IR document abstract page• Article abstract page for

journals we publish

Roll-out challenges

Who creates profiles? Who edits?

What information should be included in profiles? Who can view them?

Separate data gathering from D-Scholarship deposits?

Who promotes the service? Who trains?

Timing . . .

Future plans

Data checking

Additional data gathering

Record merging/deduping

Ability to edit user profiles and artifact records locally

Open API To allow integration with other online systems

More exhaustive scholarly practices survey for all faculty

Rollout to all Pitt Researchers Will use automatic feed from Pitt IR to PlumX

Discussion

How would you “sell” PlumX to additional faculty?