Practical Applications of Do-it-Yourself Citation Analysis

Steve BlackThe College of Saint Rose

June 10, 2012

Accompanying handout: http://www.slideshare.net/NASIG/blackpractical-applications-of-citation-analysis

Citation analysis is

the study of patterns and frequencies of citations

Why

• Objective, quantitative measurement of impact (relative frequency of citation) of – journals– authors– institutions

• Citation analysis can help answer“What are the best journals for ____?”

Critiques of relying on impact factors

• data errors• Matthew effect• gaming the system• impact vs. quality vs. importance• global vs. local impact

(See bibliography)

Never forget, & remind others when needed,

impact factor is but one of many considerations

for collection development, paper submissions,

tenure & promotion.

Do-It-Yourself Citation Analysis

• tailored to specific needs not covered elsewhere

• can be quick-and-dirty, exhaustive, or in between

• saves expense of Scopus or Web of Knowledge

Basic DIY Approach

• Select a target population– journals in a sub-discipline (e.g. forensic psychology)

– researchers at an institution, school, department, or even one faculty member (e.g. all the papers published in last 10 years by faculty of Dept. X of Big U.)

– subjects or keywords (e.g. phrases on hot research topics)

• Select a sample that represents the target population

• Compile works cited in the chosen sample• Sort and count the works cited

Google Scholar & Zotero

• Google Scholar is good for identifying articles on a topic, BUT– no authority control (which S Black?)– works cited displayed in many formats, if available

at all

Freeware can help compile . . .

. . .but Zotero’s not designed for DIY citation analysis

• emphasis on full text searching• no authority control• inconsistent data structure• extracting and downloading the right data to a

spreadsheet not straightforward

http://kimmonsdesign.com/node/24

Case study in DIY citation analysis: forensic psychology

Published as “Frequently Cited Journals in Forensic Psychology,” Psychological Reports, v.110, no.1, (2012), p.276-282.

Goal: create ranked list of most-cited journals in forensic psychology

• General Method– identify leading journals specific to the field to use

as sources of citations– gather citations from recent volumes of those

journals– compile cites and count times cited– rank cited journals by frequency of citation

Source Journals Sample

• For published article: – 2007, 2008, and 2009 volumes of 6 widely held

journals specific to forensic psychology– n=16,518 works cited

• For this presentation: – 2011 volume of Law and Human Behavior– n=1378 works cited

Tools used

• WorldCat (to identify most widely held journals)

• PsycINFO via EBSCOhost (to identify works cited)

• RefWorks (to organize works cited)

• Excel (to count and rank cited journals)

Retrieve records for articles in source journals, one issue at a time

Add cited references to folder

Export references . . .

. . . to RefWorks

RefWorks creates a temporary folder of the downloads

Create a folder for each issue’s works cited (keep discrete units to allow statistical analysis of reliability)

Move the downloaded cites to the appropriate folder

Export references in tab delimited format

Save as a text document

Open the text file in Excel . . .

. . .by going through 3 steps to convert from text

Work with cited journal titles shown as “Periodical Full”

Note that some tagged reference types are imprecise, don’t count the “slop.”

Sort the title column alphabetically

For accurate counts, review titles and remove discrepancies

Count citation frequency(choose a cutoff point equal to about 0.5% of sample)

Record frequency counts

Sort by largest to smallest to create ranked list

Bradford distribution 1:n:n2:n3 roughly holds in this case as 1:3:9:27...

Law and Human Behavior 154Criminal Justice and Behavior 45Journal of Applied Psychology 45Behavioral Sciences and the Law 40 130Journal of Personality and Social Psychology 35Psychological Assessment 32Applied Cognitive Psychology 29American Psychologist 26Psychiatric Services 26Journal of Abnormal Psychology 21Psychology, Public Policy, and Law 21Personality and Individual Differences 15Psychological Bulletin 15 220Assessment 14Psychological Review 14Journal of Applied Social Psychology 13Journal of Experimental Psychology: Applied 13Psychological Science 13Psychology, Crime and Law 12International journal of law and psychiatry 11Legal and Criminological Psychology 11Criminology 10Sex Roles 9Aggressive Behavior 8American Journal of Public Health 8International Journal of Forensic Mental Health 8International Journal of Offender Therapy and Comparative Criminology 8Journal of Criminal Law and Criminology 7Journal of personality assessment 7Journal of the American Academy of Psychiatry and the Law 7Aggression and Violent Behavior 6American Journal of Psychiatry 6Archives of General Psychiatry 6Basic and Applied Social Psychology 6British Journal of Psychiatry 6Journal of personality disorders 6Professional Psychology: Research and Practice 6Virginia Law Review 6Annual Review of Psychology 5British Journal of Criminology 5 231

...even though LHB cites a mix of forensic and general psych journals

Law and Human Behavior 154Criminal Justice and Behavior 45Journal of Applied Psychology 45Behavioral Sciences and the Law 40 130Journal of Personality and Social Psychology 35Psychological Assessment 32Applied Cognitive Psychology 29American Psychologist 26Psychiatric Services 26Journal of Abnormal Psychology 21Psychology, Public Policy, and Law 21Personality and Individual Differences 15Psychological Bulletin 15 220

How should we take this neatly ranked list of forensic psych journals?

It depends on the reliability of our rankings

• Measure reliability 2 ways:– Macro: variability in ranked lists– Micro: variability in individual titles’ times cited

Macro: How well do ranked lists correlate?

Law and human behavior 28Journal of Applied Psychology 13Psychological assessment 11Applied Cognitive Psychology 7Journal of abnormal psychology 6Assessment 5Virginia Law Review 5Criminal Justice and Behavior 4Criminology 4Wisconsin Law Review 4

Feb 2011

Law and human behavior 30Psychiatric Services 14Behavioral Sciences and the Law 12Journal of Applied Psychology 9Psychology, Public Policy, and Law 9Criminal Justice and Behavior 7Journal of personality and social psychology 6Sex Roles 6American Psychologist 5International Journal of Forensic Mental Health 5

April 2011Law and human behavior 17Criminal Justice and Behavior 14Journal of personality and social psychology 11Personality and Individual Differences 8Behavioral Sciences and the Law 6Criminal Justice Policy Review 6International Journal of Offender Therapy and Comparative Criminology 6Assessment 5Journal of consulting and clinical psychology 5Journal of Applied Social Psychology 4

June 2011

?

Micro: How variable are frequencies of citation to LHB, J App Psych, etc.?

Macro method:Test consistency (reliability) with Spearman’s rho rank correlation

coefficientrs =

D = differences in ranksN = number of ranked items

To calculate rs, first assign ranks

• Choose a cutoff so number of ranks in each group is equal.

• Handle ties by averaging, e.g. tied for 2 & 3 are each ranked 2.5.

• Assign bottom rank to titles with no match, e.g. if n=22, unmatched titles are ranked 22.

Ranks of journals cited in 2011 volume of Law and Human Behavior

(blanks will be filled with rank = 22)

Title cited in LHB 2011 frequency allAllrank FebRank AprilRank JunRank AugRank OctRank DecRankLaw and Human Behavior 154 1 1 1 1 1 1 1Criminal Justice and Behavior 45 2.5 9.5 6 2 2 9.5Journal of Applied Psychology 45 2.5 2 4.5 3 4.5 4Behavioral Sciences and the Law 40 4 3 6 4 2 16Journal of Personality and Social Psychology 35 5 16.5 7.5 3 8.5 9.5 11.5Psychological Assessment 32 6 3 13 8.5 11 4Applied Cognitive Psychology 29 7 4 17.5 13 12 13.5 2American Psychologist 26 8.5 16.5 9.5 8.5 9.5 8.5Psychiatric Services 26 8.5 2 16.5 6Journal of Abnormal Psychology 21 10.5 5 13 9.5 11.5Psychology, Public Policy, and Law 21 10.5 4.5 4.5 16Journal of Consulting and Clinical Psychology 18 12 8.5 5.5 21Personality and Individual Differences 15 13.5 4 16Psychological Bulletin 15 13.5 11.5 13 8.5Assessment 14 15.5 6.5 8.5 21Psychological Review 14 15.5 3 21Journal of Applied Social Psychology 13 18 16.5 17.5 10Journal of Experimental Psychology: Applied 13 18 16.5 8.5Psychological Science 13 18 16.5 11.5Psychology, Crime and Law 12 20 4International journal of law and psychiatry 11 21.5 16.5 13.5Legal and Criminological Psychology 11 21.5 17.5 13 16

Rank correlations among cites in single issues of Law and Human Behavior are

weak and inconsistent 2011 Feb Apr June Aug Oct Dec

2011 1

Feb 0.24 1

Apr 0.47 -0.34 1

June 0.26 -0.05 -0.2 1

Aug 0.63 0.05 0.4 0.15 1

Oct 0.59 -0.05 0.46 -0.47 0.11 1

Dec 0.23 0.25 -0.66 -0.42 -0.03 -0.02 1

How does LHB (2011) correlate with the published list?

"Frequently Cited Journals in Forensic Psychology," Psychological Reports v.110, no. 1, Feb. 2012: 276–282Title Percent of citations times cited (n=16,518) rankLaw and Human Behavior 5.71% 1243 1Behavioral Sciences and the Law 2.46% 536 2Criminal Justice and Behavior 1.79% 390 3British Journal of Psychiatry 1.63% 354 4Journal of Applied Psychology 1.62% 352 5Journal of Personality and Social Psychology 1.50% 326 6American Journal of Psychiatry 1.34% 291 7Journal of Forensic Psychiatry and Psychology 1.30% 283 8Applied Cognitive Psychology 1.20% 261 9Psychology, Public Policy, and Law 1.15% 250 10Psychology, Crime, and Law 1.08% 236 11Psychological Assessment 1.03% 224 12Personality and Individual Differences 1.01% 221 13American Psychologist 0.99% 215 14Legal and Criminological Psychology 0.96% 210 15Journal of Consulting and Clinical Psychology 0.94% 204 16Journal of the American Academy of Psychiatry and the Law 0.92% 201 17.5Sexual Abuse: A Journal of Research and Treatment 0.92% 201 17.5Archives of General Psychiatry 0.91% 198 19Psychiatric Services 0.85% 185 20Journal of Applied Social Psychology 0.84% 183 21Psychological Bulletin 0.83% 180 22Journal of Abnormal Psychology 0.70% 153 23

Spearman’s rho rank correlation for one volume of LHB vs. the larger sample is rs=.63

title

LHB 2011 rank

Psych Reports

rankD D2

Law and Human Behavior 1 1 0 0Criminal Justice and Behavior 2.5 3 -0.5 0.25Journal of Applied Psychology 2.5 5 -2.5 6.25Behavioral Sciences and the Law 4 2 2 4Journal of Personality and Social Psychology 5 6 -1 1Psychological Assessment 6 12 -6 36Applied Cognitive Psychology 7 9 -2 4American Psychologist 8.5 14 -5.5 30.25Psychiatric Services 8.5 20 -11.5 132.25Journal of Abnormal Psychology 10.5 22 -11.5 132.25Psychology, Public Policy, and Law 10.5 10 0.5 0.25Journal of Consulting and Clinical Psychology

12 16 -4 16

Personality and Individual Differences 13.5 13 0.5 0.25Psychological Bulletin 13.5 22 -8.5 72.25Assessment 15.5 22 -6.5 42.25Psychological Review 15.5 22 -6.5 42.25Journal of Applied Social Psychology 18 21 -3 9Journal of Experimental Psychology: Applied 18 22 -4 16Psychological Science 18 22 -4 16Psychology, Crime and Law 20 11 9 81International journal of law and psychiatry 21.5 22 -0.5 0.25Legal and Criminological Psychology 21.5 15 6.5 42.25

sum of D squared 684

rs =1-(6*684)/22*(222-1)

= 1 – (4104/10626) = 1- .37 rs = .63

Sample size and expected reliability

Sample sample size (works cited)

expected correlation with population

one issue of journal n=250 very low

one volume of journal n=1500 moderate

several volumes of 3-5 journals

n>10,000 strong

Micro Method

• Measure coefficients of variation for individual titles’ frequency of times cited

Coefficient of variation = standard deviation ÷ mean

Examples for LHB 2011

Times Cited in 2011 Law and Human Behavior Feb Apr June Aug Oct Dec σX mean Coeff. Variation

Law and Human Behavior 28 30 17 25 27 27 4.55 25.67 18%Journal of Applied Psychology 13 9 0 8 8 7 4.23 7.50 56%Psychology, Public Policy, and Law 1 9 0 0 8 3 4.04 3.50 115%

Greater Coefficient of Variation = lower reliability

• Top journal pretty stable• Titles lower down the ranked list are more variable

Assuming the sampled journals validly represent the topic,

a sample of

• n<1,000 works cited can indicate the top journal (or tell you if there is none)

• n>1,000 can generate a rough indication of leading journals

• n>10,000 can create a useful ranked list

• Even with n>10,000, different samples will yield different ranks, especially further down the list,

• So a journal’s rank must ALWAYS be taken as an approximation of its “true” ranking.

Strengths of DIY Citation Analysis

• Can target a precise area• Yields data not otherwise available• Objective, quantitative collection development tool• Results may be of fairly broad interest

(publishable!)• If project is well organized, data can be gathered &

sorted by research assistant(s)• Interdisciplinary--can team with faculty outside the

library

Weaknesses of DIY Citation Analysis

• Very time consuming to do well• A really robust ranking requires n>20,000, and

even that’s not definitive• Works cited may not be readily available (few

databases make it as easy as PsycINFO)• It’s recreating the (very expensive) wheel –

better to use Scopus or Web of Knowledge if you can

Tips for publication• Don’t be too parochial or narrow—editors must think their readers

will be interested.• BUT choose something not already in Journal Citation Reports• Be very thorough with your lit review before gathering data (search

disciplinary databases as well as the library literature)• Gather citations and test sample before solidifying your method• Carefully select and describe the sample

– representative– clearly defined– robust– good face validity

• Group data and run statistics by journal volume• Consider submitting to a journal in the topic area

Tips for Collection Development

• Use citation analysis to assess impact of specialized journals

• Don’t be surprised to find surprises

• Judge new titles by other criteria

• Use ranked lists for assessment evidence

Practical Applications

• What’s the top journal on this topic?– Compile cites from 10-20 articles on the topic to identify it (or

find there’s not one)• What does the library need to support a proposed new

major?– Compile cites from 1-2 volumes of 2-4 journals to flag titles not

already available• What’s that herd of cats in Dept. X REALLY use?

– Analyze publications & dissertations from last decade• OMG I’m up for tenure soon!

– Identify a topic and do a thorough analysis of 2-3 volumes of 4-6 journals; include reliability tests

Steve Blackblacks@strose.edu

(518) 458-5494