Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact John Barnett Scholarly...
-
Upload
tobias-byrd -
Category
Documents
-
view
222 -
download
0
Transcript of Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact John Barnett Scholarly...
Here Comes the Sunburst:Measuring and Visualizing Scholarly Impact
John Barnett
Scholarly Communications Librarian
Jennifer Chan
Assistant Scholarly Communications Librarian
Office of Scholarly Communication and PublishingUniversity Library System
University of Pittsburgh
University of Pittsburgh
Pittsburgh campus + regional campuses in Bradford, Greensburg, Johnstown, and Titusville
16 undergraduate, graduate, and professional schools
456+ degree programs 2012: conferred 8,949 degrees
University of Pittsburgh
Top 10 American higher ed. in federal funding (NSF)
Top 5 in annual research support (NIH)
5,369 faculty; 4,470 full-time faculty Research conducted: more than 300
centers, institutes, laboratories, clinics
University Library System
ARL
22nd largest academic library system in North America
25 libraries; 6.6 million volumes
279,000 current serials
Sum of the Parts
Liaison Librarians
ULS Department of
Information Technology
Office of Scholarly
Communication and Publishing
Why Pitt?
Strategic goal: Innovation in scholarly communication
Providing services that scholars understand, need, and value
Putting ourselves in faculty “spaces”
Re-envisioning our librarian liaison program
Deepening our understanding of scholarly communications issues
Why PlumX? Making research “more assessable
and accessible”– Gathering information in one place– Making it intelligible and useful
Measuring and visualizing research impact
Correlating metrics from traditional and new forms of scholarly communication
Allowing researchers, labs, departments, institutions to track real-time scholarly impact
Promoting research, comparing with peers, connecting with new research
Altmetrics Project Timeline
Spring 2012: • First meeting with
Plum Analytics
Fall 2012• Gathered data
from pilot participants
Spring 2013• Faculty surveyed;
enhancements made
Pilot project aims
Develop a tool for measuring and visualizing research impact
Gathering information in one place
Intelligible and useful
Impact in social media and other scholarly communication methods
Traditional measures counted as well
See where to disseminate works to increase impact
Traditional vs. new
•Traditional measures are also counted
•Findings are complementary to conventional methods of measuring research impact (e.g., H-Index)
•Not intended to replace them
New measures
More comprehensive: Altmetrics = ALL METRICS– Citations– Usage– Captures– Mentions– Social Media
Covers impact of online behavior– Because scholars increasingly work online
Measures impact immediately– Because citation counts take years to appear in literature
Pilot Process
Created Altmetrics Task Force
Engaged Liaison Librarians
CV receipt and records creation
Pilot Project Participants
• 32 researchers, various disciplines
• 9 schools
• 18 departments
• 1 complete research group
• Others joined as they learned about the project
Pilot Project Participantsdiscipline
online behavior
level of career advancement
Selected faculty
participants, diversified by:
Technologies
Internal– IR built on Eprints Platform– Sharepoint– Microsoft Office Suite– PMID/DOI data import tool
External– PlumX– DOIs– PMID
Data collection for pilot project
• Created records in D-Scholarship@Pitt, our institutional repository
• Focused on articles, books, book chapters, proceedings
• Scholarly output with standard identifiers• DOI, ISBN, PubMed
ID, official URL, etc.
• Scholarship produced since 2000
Other Library work
• Developed guidelines to standardize record creation
• Data entry from faculty CVs into IR (2 to 3 student workers with QA by librarians)
• Librarian liaisons and other staff trained in record creation
• SharePoint site used to track work completed
• Coordination with pilot faculty
• Gathered feedback and administered online survey
Metadata sources
Faculty CVs . . . But verify metadata!
Books: PittCat, WorldCat, Books in Print, publisher sites, online retailers
Journals: Serials Solutions list, journal websites, JournalSeek, UlrichsWeb, DOAJ, PubMed
Conference presentations: Websites, PittCat, indexes, WorldCat
PMID Import Tool
Custom build by SysAdmin for Eprints Platform
Utilizing PMIDs from PubMed, able to import records that prepopulate metadata fields– Item Type, Title, Abstract, Creators, Publication Title, ISSN,
Volume/Issue, Page ranges, Date and Date type, DOI, MeSH Headings, Grant Information, Keywords, etc.
Full-text sources
DOAJ
ERIC
PLOS
SSRN*
Other repositories*
Federal government websites*
Conference websites*
* Use with caution
Plum Analytics processing activities
Harvest records from Pitt IR for each participant
Harvest additional online artifacts NOT in Pitt IR
Use data mining to harvest publically available metrics from hundreds of sites on the Web
C
r
e
a
t
e
v
i
s
u
a
li
z
a
ti
o
n
s
t
o
d
i
s
p
l
a
y
m
e
tri
c
s
o
n
P
l
u
m
X
i
n
t
e
rf
a
c
e
Key features
Faculty profiles
Online ‘artifacts’– Article– Book– Book chapter– Video– Etc.
Impact graph
Sunburst
Feedback
• Solicited via email and online survey
• Generally positive in most cases
• Data corrections• Errors in profiles• Links to wrong data• Quickly corrected by Plum staff
• Requests for results from additional online sources (Google Scholar, SlideShare, Reddit, etc.)
• PlumX collects data from these but did not gather information in advance for profiles
The survey says
Surveyed pilot project faculty in spring 2013
@ 1/3rd responded to the survey
Meaning 13 out of 32 participants responded
Accurate and useful data
Summary profile data page0
10
20
30
40
50
60
70
80
90
15.38
7.69
76.92
0
Strongly disagreeDisagreeAgreeStrongly agree
The bar graph
Usefulness of interactive bar graph0
10
20
30
40
50
60
70
25
58.33
16.67
Strongly disagreeDisagreeAgreeStrongly agree
0
The sunburst
Usefulness of sunburst0
10
20
30
40
50
60
41.67
50
8.33
Strongly disagreeDisagreeAgreeStrongly agree
0
Traditional & new measures
Conveying traditional and new measures0
10
20
30
40
50
60
36.36
54.55
9.09
Strongly disagreeDisagreeAgreeStrongly agree
0
Usefulness of altmetrics
The value of altmetrics0
10
20
30
40
50
60
18.18 18.18
54.55
9.09
Strongly disagreeDisagreeAgreeStrongly agree
Learning something new
Learning something new about my own research0
5
10
15
20
25
30
35
40
9.09
36.36
27.27 27.27
Series 1Series 2Series 3Series 4
Comments
Affiliations/bio inaccurate or has missing information
“Mentions” by whom & when?
Publications misclassified– Books vs. conference proceedings
Data not collected– Google Scholar– Slideshare
Comments
Filter out unwanted information
Data are wrong—and not useful
Overabundance of information in sunburst
“I only care what a select group of scholars thinks of my work”
“I did not find this useful for my discipline”
Observations
Lacked information about faculty practices
Are the results useful to all faculty, all disciplines?
May appeal more to faculty who are early in their careers or whose work is more contemporary
Will the data be used against faculty or programs?
Labor-intensive strategy
When it comes down to it . . . Does anyone care?
Embeddable widgets(in development)
For researchers, to add to:• their own Web pages• department directories• IR researcher profile page
For individual artifacts,to build article level metrics for imbedding in:
• IR document abstract page• Article abstract page for
journals we publish
Roll-out challenges
Who creates profiles? Who edits?
What information should be included in profiles? Who can view them?
Separate data gathering from D-Scholarship deposits?
Who promotes the service? Who trains?
Timing . . .
Future plans
Data checking
Additional data gathering
Record merging/deduping
Ability to edit user profiles and artifact records locally
Open API To allow integration with other online systems
More exhaustive scholarly practices survey for all faculty
Rollout to all Pitt Researchers Will use automatic feed from Pitt IR to PlumX