Beyond Citation Searching: new tools to measure research impact

Elaine Lasda Bergmanelasdabergman@albany.edu

Professional Activities CommitteeInvited Presentation

January 21, 2016

Overview• Bibliometrics– Origins– Tools• WoS• Scopus• GS

– Issues

Overview• Other Models

• Becker Model– Altmetrics–Open Science

• Takeaways

Photo source: http://upload.wikimedia.org/wikipedia/commons/thumb/1/18/%22Citation_needed%22.jpg/1024px-%22Citation_needed%22.jpg

What is a “metric?”•  4. A system or standard of measurement; a criterion or set of criteria stated in

quantifiable terms.• 1934   H. C. WARREN Dict. Psychol. 166/2   Metric, a system or standard of measurement in

terms of which the conclusions stated hold.• 1951   J. S. BRUNER in R. R. Blake & G. V. Ramsey Perception v. 141   Our basic metric will

involve the comparison of group scores.• 1968   Language 44 715   The simplicity metric demands that one choose the unmarked

segment as the underlying one, since its choice leads to less complexity in the phonological representation.

• 1973   A. H. SOMMERSTEIN Sound Pattern Anc. Greek iii. 94   This statement does not depend on the acceptance of a feature-counting simplicity metric. P. H. Matthews, in a seminar at Cambridge, has criticized the assumption that such a metric is the appropriate evaluation measure for grammars.

• 1993   Computing 19 Aug. 21/3   ‘You can find relevant metrics for all key requirements,’ he says. ‘Ease of learning, for example, can be measured by setting targets for novice users that are a high percentage of the known performance of experts.’

• "metric, n.1 and adj.1". OED Online. December 2015. Oxford University Press. http://www.oed.com/view/Entry/117657?result=1&rskey=bAcR8e& (accessed January 11, 2016).

Origin of the term Bibliometrics

Pritchard, A. (1969). Statistical bibliography or bibliometrics. Journal of Documentation, 25, 348-9.

• Citation count• Impact Factor• Immediacy Index• Citation Half-Life

Origins of Bibliometrics

Photo: http://www.chemheritage.org/Discover/Collections/Oral-Histories/Details/asset_upload_file66_69008_thumbnail.jpg

• Citation count• Impact Factor• 5-year Impact Factor• Immediacy Index• Number of Articles• Cited Half-life• Eigenfactor Score• Article Influence Score

Journal Impact Factor

• Strengths• Limitations

Number of citations to a journal in a given year from articles occurring in the past 2 years

Divided by the number of scholarly articles published in the journal in the past 2 years

InCites Platformhttp://wokinfo.com/products_tools/analytical/jcr/

Cited Reference Searching

Citing vs. Cited References

• Citing References – scholarly journal (or scholarly book) content

indexed in WoS – the articles that CITE the cited references

• Cited References– Any (most) references cited in a scholarly journal

article indexed in WoS– Can be items that are not themselves indexed

in WoS IF an indexed journal article cited it• Example: newspaper articles

Core Collection• Contents

– 12K “highest impact” journals– 160K conference proceedings

– Science Citation Index Expanded (SCI-EXPANDED) --1900-present Social Sciences Citation Index (SSCI)

– Social Sciences Citation Index (SSCI) --1900-present Arts & Humanities Citation Index (A&HCI)

– Arts & Humanities Citation Index (A&HCI) --1975-present Conference Proceedings Citation Index- Science (CPCI-S)

– Emerging Sources Citation Index (ESCI) -- 2015-present– Conference Proceedings Citation Index- Science (CPCI-S) --1990-present– Conference Proceedings Citation Index- Social Science & Humanities (CPCI-SSH)

--1990-present– Book Citation Index– Science (BKCI-S) --2005-present– Book Citation Index– Social Sciences & Humanities (BKCI-SSH) --2005-present– Current Chemical Reactions (CCR-EXPANDED) --1985-present– Includes Institut National de la Propriete Industrielle structure data back to 1840– Index Chemicus (IC) --1993-present

Book Citation Index• Contents– Scholarly books & chapters (count??)– NOT• Textbooks • Encyclopedias/ Reference Books

– 2005 to present

Data Citation Index• Contents (counts?)– Some pre-1900 but most 1900 to

present– Repositories– Data Studies– Data Sets– Software

Exampleshttp://library.albany.edu/db/resource/?id=4158

Scopus• Content– 22K titles

• 20.8K Peer reviewed journals• 3,850 Journal “articles in press”• 367 Trade journals• 400 Book series• 40K Books• 6.4M Conference papers

– 21M records from 1823-1996– 25.2 M patent records

Journal Metrics• SCImago Journal Rank• Impact Per Publication• Source Normalized Impact per Paper• Citation Count• Document count per year• Percent never cited • Percent review articles

Article Metrics• Citation Count & percentile benchmark• Field Weighted Citation Impact• Mendeley Readership Count & percentile

benchmark• Count of scholarly commentary– Blogs, Wikipedia

• Count & benchmark of social activity– Twitter, Facebook

• Others, broken down by source

Author Metrics• h-index & h-graph• Citation overview tracker• Analysis & Viz of author output

Cited Reference Searching

• Use Author or Author Identifier search

• Limit by Affiliation if you can• DIRTY DATA IS HARDER TO FIND IN

SCOPUS!• Examples

http://library.albany.edu/db/resource/?id=4018

Google Scholar Content

They do, however, try to help

https://scholar.google.com/intl/us/scholar/about.html

Guidelines for Authors

Guidelines for Publishers

Guidelines for Repositories

Cited Reference Searching in GS (the Hard Way)

Examplehttp://scholar.google.com

Publish or Perish• Downloadable tool• Metrics:

– Total # of papers– Total # of citations– h-index– g-index– Contemporary h-index– Still other h-index variations– Average annual increase in h-index– Age weighted citation rate– Authors per paper metrics

http://www.harzing.com/

Author Profiles (Google Scholar Citations)

Impact Factor Scams

http://scholarlyoa.com/2013/08/06/bogus-impact-factor-companies/

Academia Fights Back!

http://www.theguardian.com/science/political-science/2013/may/17/science-policy

Becker Modelhttps://becker.wustl.edu/impact-assessment

• Strengths • Limitations

QUESTION:

WHAT CONSTITUTES SCHOLARLY OUTPUT???

Altmetrics• Altmetrics Manifesto

– Peer review is “slow”– Existing metrics “insufficient”– New tools provide new ways to measure

• Zotero• Mendeley• Twitter

– Scholarly output is NOT merely peer reviewed journal articles• “raw science”• nanopublication• self-publishing

J. Priem, D. Taraborelli, P. Groth, C. Neylon (2010), Altmetrics: A manifesto, 26 October 2010. http://altmetrics.org/manifesto

Altmetrics Tools• Altmetric.com (bookmarklet)

– http://www.altmetric.com/

• Microsoft Academic (author, institution profiles)– http://academic.research.microsoft.com/Default.aspx

• ImpactStory –scholar/user subscription– https://impactstory.org

• Mendeley• Scholarometer• Topsy• PlumX

Open Science/Open Access

• Open Data• Open Access• Open Source• Citizen Science– Etc. etc. etc.

Open Access• Greater visibility – no paywall • “Google-able”• Intellectual freedom• Dissemination of research• Government grant requirements• http://libguides.library.albany.edu/scholcomm

Scholars Archive (repository) Metrics

Scholars Archive (Author) Metrics

Author Dashboardhttp://scholarsarchive.library.albany.edu/cgi/myaccount.cgi?context=

Other Open Science Tools:

• VIVO – http://www.vivoweb.org/

• PeerEvaluation– http://www.peerevaluation.org/

• Kudos – https://www.growkudos.com/

Scholarly Networking• Faculty of 1000 – http://f1000.com/

• Figshare – http://figshare.com/

• GitHub– https://github.com/

• Zenodo – http://zenodo.org/

• ResearchGate• https://www.researchgate.net/home

Regular Social Media• SlideShare• Blogs• Twitter• YouTube/Vimeo• Facebook • LinkedIN

Awesome Resource!

http://blog.impactstory.org/research-impact-challenge-ebook/

Takeaways• Bibliometrics

– Increased number of metrics– Increased sources of cited and citing references– Increased formats covered

• Altmetrics– Fast– Diverse– New forms of scholarly output– New forms referencing scholarly output

• Open Access and Open Science– Increase impact and visibility– Increase quality of scientific methods and research– New measures of open scholarly output

• Not necessarily correlated with traditional bibliometrics

Questions ?