Usage Factor: Final Report & Next Steps

25
Usage Factor: Final Report & Next Steps John McDonald, Director, Libraries of the Claremont Colleges Usage Statistics: New Developments and Practical Applications NFAIS Committee on Usage Statistics New York City, October 21, 2008

description

Presentation given to NFAIS, October 21, 2008

Transcript of Usage Factor: Final Report & Next Steps

Page 1: Usage Factor: Final Report & Next Steps

Usage Factor:

Final Report & Next Steps

John McDonald, Director, Libraries of the Claremont Colleges

Usage Statistics: New Developments and Practical ApplicationsNFAIS Committee on Usage Statistics

New York City, October 21, 2008

Page 2: Usage Factor: Final Report & Next Steps

Usage Factor Genesis

COUNTER statistics provide a reliable comparison of amount of use between journals

But they don’t provide a meaningful usage-based measure of relative quality or value

ISI's Impact Factor compensates for the fact that larger journals will tend to be cited more than smaller ones

Page 3: Usage Factor: Final Report & Next Steps

Who will be interested? Is there a demand for it among:-

Publishers Authors Librarians

What are the practical issues that would need to be addressed?

Who will do it & how much will it cost to develop and maintain?

Page 4: Usage Factor: Final Report & Next Steps

Research review

From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

Page 5: Usage Factor: Final Report & Next Steps

Some initial evidence……

From New journal publishing models: an international survey of senior researchers; Ian Rowlands and Dave Nicholas, A CIBER report for the Publishers Association and the International Association of STM Publishers, 22 September 2005

Page 6: Usage Factor: Final Report & Next Steps

Addressing the challenge…….

ISI's Impact Factor provides an industry standard for journal quality.

Can we do something similar for usage?

In other words, should we seek to develop a usage-based measure of journal quality or value?

What metrics should be used?

Page 7: Usage Factor: Final Report & Next Steps

The Research…….

In 2007 UK Serials Group and COUNTER published research which examined:- the various ways in which journal quality is

currently assessed the degree to which any additional usage-

based metrics might prove valuable to each stakeholder community

practical ways in which such metrics might be derived and constructed to provide the maximum utility for all, within defined resource constraints

Page 8: Usage Factor: Final Report & Next Steps

Stage 1 Phase 1 October 2006 – January

2007 COUNTER Director Peter Shepherd carried

out a series of in-depth interviews with:- 7 authors 9 librarians 13 publishers

Phase 2 March 2007 Broader web based survey of:-

155 librarians 1400 academic authors

Results June 2007 Final report published by UKSG

Page 9: Usage Factor: Final Report & Next Steps

Stage 1, Phase 1 Results - Users

Would Journal Usage Factors be helpful to you in assessing the value, status and relevance of a journal?

Librarians – YES 100% Authors - YES 100%

Page 10: Usage Factor: Final Report & Next Steps

Phase 1 Usage factor advantages

A useful counterweight to Impact Factors

Especially helpful for journals and fields not covered by ISI

Especially helpful for journals with high undergraduate or practitioner use

Especially helpful for journals publishing relatively few articles

Data available potentially sooner than with Impact Factors

Page 11: Usage Factor: Final Report & Next Steps

Phase 1 Usage factor advantages

Authors select journals that will give their articles prestige and reach. Impact Factor is a widely used surrogate for the former, while perceived circulation and readership reflect the latter. But usage is becoming more important as a measure of reach.

Carol Tenopir

A simple, usage based metric would make usage more understandable to editors and authors as a measure of value. There is currently much talk of usage and a lot of data, which the non-librarians find confusing.

Publisher

Page 12: Usage Factor: Final Report & Next Steps

Phase 1 Issues to address

Impact Factors are well known and relied upon in the industry

Not all publishers are COUNTER compliant.

COUNTER data may not be robust enough for this.

How would print usage, still significant for many journals, be taken into account?

Would another global measure, such as usage half-life per journal or per discipline, be of greater value?

Usage based metrics could stimulate publishers to inflate their usage or be influenced by sales forces and technology investments.

Many journals publish in multiple hosting sites, making calculation of a single usage metric problematic.

Page 13: Usage Factor: Final Report & Next Steps

Phase 1 Who will construct the measure?

Publishers are, on the whole, unwilling to provide their usage data to a third party for consolidation calculation of Usage Factors.

The majority appear to be willing to calculate UFs for their own journals and to have this process audited.

This is generally perceived as a natural extension of the work already being done for COUNTER.

While it may have implications for systems, these are not seen as being problematic.

Some publishers already consolidate their usage data from several sources and also maintain figures on the total number of articles published in each of their journals.

Page 14: Usage Factor: Final Report & Next Steps

Basis for Usage Factor

Usage Factor =

Total usage over period ‘x’ of articles published during period ‘y’

÷Total articles published during period ‘y’

Page 15: Usage Factor: Final Report & Next Steps

Phase 1 Definitions

Many views on how UF components should be defined.

In particular, how to define total usage specified usage period total number of articles published online specified publication period

Tests with real usage data will be required to refine the definitions for these terms.

Page 16: Usage Factor: Final Report & Next Steps

Phase 1 What to measure

TOTAL USAGE There was agreement that usage should be

total downloads, as specified in COUNTER JR1.

SPECIFIED USAGE PERIOD There was overwhelming agreement that the

specified usage period be one calendar year.

TOTAL NUMBER OF ARTICLES PUBLISHED ONLINE Need to account for journals not covered by ISI Is the ISI definition of “source items” too narrow

when measuring usage?

Page 17: Usage Factor: Final Report & Next Steps

Phase 1 What to measure

SPECIFIED PUBLICATION PERIOD There was a diversity of responses to this

question, with no clear consensus on any time period among authors, librarians or publishers.

Insufficient data to support the selection of

any one specific option

Tests using real data will be required before any firm conclusions can be drawn.

Page 18: Usage Factor: Final Report & Next Steps

Phase 2 Librarian results: new journals

Ranking without Usage Factor Ranking with Usage Factor

1. Feedback from library users 1. Feedback from library users

2. Price 2. Usage Factor

3. Reputation/status of publisher 3.Price

4. Impact Factor 4. Impact Factor

5. Reputation/status of publisher

Page 19: Usage Factor: Final Report & Next Steps

Phase 2 Librarian results: existing journals

Ranking without Usage Factor

Ranking with Usage Factor

1. Feedback from library users 1. Feedback from library users

2. Usage 2. Usage

3. Price 3. Usage Factor

4. Cost per Download 4. Price

5. Impact Factor 5. Cost per Download

6. Reputation/status of publisher

6. Impact Factor

7. Reputation/status of Publisher

“I would view Usage Factor as an aid for collection rather than cancellation decisions. Usage per se is a more suitable tool for us when considering cancellation.”

Page 20: Usage Factor: Final Report & Next Steps

Stage 1 recommendations

UF be developed to the point of testing as a practical, implementable measure of journal quality, value and status.

On May 18, 2007 the UKSG Committee accepted these recommendations.

Page 21: Usage Factor: Final Report & Next Steps

Next Steps: Stage 2

Test each of the individual elements in the UF equation using real publisher usage data.

Compare UF ranks with IF ranks.

Refine and further define the workflow/organization scenarios for the definition, calculation and dissemination of the Usage Factor.

Page 22: Usage Factor: Final Report & Next Steps

Stage 2 – The Plan

Project Steering Group established (6 publishers, 1 aggregator, 1 hosting service, wider geographical library representation

Usage logs to be converted to uniform standard report format for analysis by expert third party

RFP currently in DRAFT form for third party selection process

Page 23: Usage Factor: Final Report & Next Steps

Stage 2 – The Deliverables A report (early 2009) which will:

Outline the various metrics assessed

Recommend which of them prove consistent and robust enough to be adopted for scaled up onward monitoring

Suggest any ways in which data providers might amend the way they capture, structure, label, and maintain their data which would make the measurement of Usage Factors easier and more reliable.

Propose ways to audit Usage Factors for accuracy

Page 24: Usage Factor: Final Report & Next Steps

Importance….. “Currently journal publishers are under

a lot of pressure to demonstrate the value they provide. By participating in this process, publishers will influence it, helping to develop useful measures in which they can have confidence.”

“This is going to happen in any event, so it is best that UF is developed and implemented by a trusted organization in which publishers are represented.”

Page 25: Usage Factor: Final Report & Next Steps

Usage Factor Project

More information at: http://www.uksg.org/usagefactors

Contact: Richard Gedye, Project Director

[email protected]