Usage Factor:

Final Report & Next Steps

John McDonald, Director, Libraries of the Claremont Colleges

Usage Statistics: New Developments and Practical ApplicationsNFAIS Committee on Usage Statistics

New York City, October 21, 2008

Usage Factor Genesis

COUNTER statistics provide a reliable comparison of amount of use between journals

But they don’t provide a meaningful usage-based measure of relative quality or value

ISI's Impact Factor compensates for the fact that larger journals will tend to be cited more than smaller ones

Who will be interested? Is there a demand for it among:-

Publishers Authors Librarians

What are the practical issues that would need to be addressed?

Who will do it & how much will it cost to develop and maintain?

Research review

From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

Some initial evidence……

From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

Addressing the challenge…….

ISI's Impact Factor provides an industry standard for journal quality.

Can we do something similar for usage?

In other words, should we seek to develop a usage-based measure of journal quality or value?

What metrics should be used?

The Research…….

In 2007 UK Serials Group and COUNTER published research which examined:- the various ways in which journal quality is

currently assessed the degree to which any additional usage-

based metrics might prove valuable to each stakeholder community

practical ways in which such metrics might be derived and constructed to provide the maximum utility for all, within defined resource constraints

Stage 1 Phase 1 October 2006 – January

2007 COUNTER Director Peter Shepherd carried

out a series of in-depth interviews with:- 7 authors 9 librarians 13 publishers

Phase 2 March 2007 Broader web based survey of:-

155 librarians 1400 academic authors

Results June 2007 Final report published by UKSG

Stage 1, Phase 1 Results - Users

Would Journal Usage Factors be helpful to you in assessing the value, status and relevance of a journal?

Librarians – YES 100% Authors - YES 100%

Phase 1 Usage factor advantages

A useful counterweight to Impact Factors

Especially helpful for journals and fields not covered by ISI

Especially helpful for journals with high undergraduate or practitioner use

Especially helpful for journals publishing relatively few articles

Data available potentially sooner than with Impact Factors

Phase 1 Usage factor advantages

Authors select journals that will give their articles prestige and reach. Impact Factor is a widely used surrogate for the former, while perceived circulation and readership reflect the latter. But usage is becoming more important as a measure of reach.

Carol Tenopir

A simple, usage based metric would make usage more understandable to editors and authors as a measure of value. There is currently much talk of usage and a lot of data, which the non-librarians find confusing.

Publisher

Phase 1 Issues to address

Impact Factors are well known and relied upon in the industry

Not all publishers are COUNTER compliant.

COUNTER data may not be robust enough for this.

How would print usage, still significant for many journals, be taken into account?

Would another global measure, such as usage half-life per journal or per discipline, be of greater value?

Usage based metrics could stimulate publishers to inflate their usage or be influenced by sales forces and technology investments.

Many journals publish in multiple hosting sites, making calculation of a single usage metric problematic.

Phase 1 Who will construct the measure?

Publishers are, on the whole, unwilling to provide their usage data to a third party for consolidation calculation of Usage Factors.

The majority appear to be willing to calculate UFs for their own journals and to have this process audited.

This is generally perceived as a natural extension of the work already being done for COUNTER.

While it may have implications for systems, these are not seen as being problematic.

Some publishers already consolidate their usage data from several sources and also maintain figures on the total number of articles published in each of their journals.

Basis for Usage Factor

Usage Factor =

Total usage over period ‘x’ of articles published during period ‘y’

÷Total articles published during period ‘y’

Phase 1 Definitions

Many views on how UF components should be defined.

In particular, how to define total usage specified usage period total number of articles published online specified publication period

Tests with real usage data will be required to refine the definitions for these terms.

Phase 1 What to measure

TOTAL USAGE There was agreement that usage should be

total downloads, as specified in COUNTER JR1.

SPECIFIED USAGE PERIOD There was overwhelming agreement that the

specified usage period be one calendar year.

TOTAL NUMBER OF ARTICLES PUBLISHED ONLINE Need to account for journals not covered by ISI Is the ISI definition of “source items” too narrow

when measuring usage?

Phase 1 What to measure

SPECIFIED PUBLICATION PERIOD There was a diversity of responses to this

question, with no clear consensus on any time period among authors, librarians or publishers.

Insufficient data to support the selection of

any one specific option

Tests using real data will be required before any firm conclusions can be drawn.

Phase 2 Librarian results: new journals

Ranking without Usage Factor Ranking with Usage Factor

1. Feedback from library users 1. Feedback from library users

2. Price 2. Usage Factor

3. Reputation/status of publisher 3.Price

4. Impact Factor 4. Impact Factor

5. Reputation/status of publisher

Phase 2 Librarian results: existing journals

Ranking without Usage Factor

Ranking with Usage Factor

1. Feedback from library users 1. Feedback from library users

2. Usage 2. Usage

3. Price 3. Usage Factor

4. Cost per Download 4. Price

5. Impact Factor 5. Cost per Download

6. Reputation/status of publisher

6. Impact Factor

7. Reputation/status of Publisher

“I would view Usage Factor as an aid for collection rather than cancellation decisions. Usage per se is a more suitable tool for us when considering cancellation.”

Stage 1 recommendations

UF be developed to the point of testing as a practical, implementable measure of journal quality, value and status.

On May 18, 2007 the UKSG Committee accepted these recommendations.

Next Steps: Stage 2

Test each of the individual elements in the UF equation using real publisher usage data.

Compare UF ranks with IF ranks.

Refine and further define the workflow/organization scenarios for the definition, calculation and dissemination of the Usage Factor.

Stage 2 – The Plan

Project Steering Group established (6 publishers, 1 aggregator, 1 hosting service, wider geographical library representation

Usage logs to be converted to uniform standard report format for analysis by expert third party

RFP currently in DRAFT form for third party selection process

Stage 2 – The Deliverables A report (early 2009) which will:

Outline the various metrics assessed

Recommend which of them prove consistent and robust enough to be adopted for scaled up onward monitoring

Suggest any ways in which data providers might amend the way they capture, structure, label, and maintain their data which would make the measurement of Usage Factors easier and more reliable.

Propose ways to audit Usage Factors for accuracy

Importance….. “Currently journal publishers are under

a lot of pressure to demonstrate the value they provide. By participating in this process, publishers will influence it, helping to develop useful measures in which they can have confidence.”

“This is going to happen in any event, so it is best that UF is developed and implemented by a trusted organization in which publishers are represented.”

Usage Factor Project

More information at: http://www.uksg.org/usagefactors

Contact: Richard Gedye, Project Director

richard.gedye@oxfordjournals.org