Defining New Metrics For Libraries Success iSchool, University of Toronto Symposium April 23, 2014...

Defining New Metrics For Libraries SuccessiSchool, University of Toronto Symposium

April 23, 2014

Dr. Bill Irwin, Huron University College

Paul St-Pierre, PHD Candidate, FIMS, UWO

In

Insights: Organizational cultures in public libraries: Gatekeepers to effective outcome evaluation?

Project’s Goal

Organizational Culture & Evaluation2

The key goal of our research is to examine existing evaluation systems of public libraries, in an effort to replace the current reliance on a predominantly output based model of performance measure, with new models of evaluation based on capturing outcomes.

Project’s Goal


Outcome models assist policy makers, both inside and outside of organizations, to address fundamental questions of resource allocation, delivery methods, and agency design and purpose.

Defining Outcome Evaluation


Project Objectives


exploring the limitations and challenges inherent in the current qualitative evaluation system

creating a better understanding of the cultural impacts and policy implications of a sociological performance framework, on library organizations, their constituencies, and their stakeholders in the establishment

capture changes in the Practice-Program-Policy continuum and the downstream implications for how libraries are viewed and valued upon the introduction of a new system of performance measurement

OPLA Child and Youth Services Committee Teen Services Benchmarks and Statistical Report 2013


Does your library measure outcomes or impacts of teen programming?

% Yes

42.9%

All 46.7%

100,000+ 38.1%

50,000<100,000

40.0%

15,000<50,000 46.4%

5,000<15,000 75.0%

<5,000 42.9%

Less than 50% of the libraries reported measuring outcomes or impacts of teen programming, althoughseveral libraries did note that they are planning to implement outcome-based measurement in the future.

Q: Are you satisfied with the methods used by your library to evaluate teen programs?


• It would be helpful to have a clearer idea of what a youth program should be i.e. what is the purpose, what is the desire outcome.

• It is not consistent• We do not evaluate

anything past attendance.• I would like to find out

what the participant got out of program

• Difficult to measure anecdotal reports

• Evaluation is based only on circulation and attendance performance indicators

Q: Are outcomes/impacts measured in other parts of your library?


• Lack of staff capacity• Do not have anyone trained

in outcome/impact measurement

• Probably the best answer for why it hasn't happened yet is "too time consuming."

• Lack of time.• Not currently considered a

priority by our governing body

• Assessing outcomes falls short of our priority list.

• No formal criteria which is implemented system-wide

Q: You indicated that you do not assess outcomes/impacts of teen programs. Why not? Check all that apply.


Q: Do the agencies that fund your teen programs require that you report outcome/impact measures?


Organizational Culture and Impact


Sub-cultures are currently the greatest inculcator of values (Schein, 1992) regarding: 1.the worth and purpose of evaluations,2.postulation of negative influence, 3.evaluation seen as “busy work”

Sub-Culture as Inhibitor


Common Responses from Follow-up Interviews

Seen as “busy-work” Rely on informal feedback Some staff value the process,

some question it, seen as additional work

Against staff comfort level Comments are filtered –each

manager chooses Staff wouldn’t want it (new

system) to be more onerous on them

Staff question why, “Are you evaluating me?”

Inhibitors


Lack of education, not able to produce meaningful results and change

Lack of inclusion: “Big picture” relevanceRequired training in best practicesLibrarians feel skill set is inadequate to the

task, so they are reluctant to engage – (Similar circumstances to the introduction of technology in mid to late 1990s in public libraries (Author’s observation))

Technological impasse Consider the law of unintended consequences

(how can/does it apply here?)

Sustainable change requires cultural focusing (1)


"Cultural change is one of the most important factors to consider but one of the hardest to implement" (Preston, 2004).

“In general, we found a number of library staff skeptical of quantitative or qualitative data from customers, preferring instead to rely on their own assumptions and past practices to make decisions.” (Hiller et al, 2008)

“Due to the ‘everyone does the same thing’ culture and operational model it was impossible to make a change to workflow in one area without it directly impacting other areas - therefore systemic change was necessary.” (Nussbaumer & Merkley, 2012)

Sustainable change requires cultural focusing (2)


Integration of Culture & Evaluation Model (1)


The following model was developed to demonstrate the steps an organization goes through in regards to moving from an input based to an outcome based evaluation system.

It highlights the operational and cultural transformation process.

This model can be used to determine where an organization is situated (at what stage) in terms of its use of outcome evaluation.

The model is designed to serve as a “roadmap”, to assist organizations in their successful movement through a series of steps, by infusing a culture of outcome evaluation.

Once an organization has achieved stage 4, evaluation then serves as a tool for: realizing organizational priorities, educating stakeholders and funders of the range of quantitative and qualitative program and service impacts, and identifying operational issues and challenges.


17

Creating an outcome evaluation culture



Kotter (2008) argues that getting buy-in is not enough because it only engages the head, not the heart.

Moving through the stages is akin to Kotter’s step 6, creating short term wins, building staff confidence, an incremental approach

Somewhere in the transition between step 1 to 2 need to create and articulate a vision/mission statement. Why do we evaluate? What is it all about?

Building cultural assessment seen as a strategic priority – otherwise over shadowed by more “important work”

“When faculty and staff perceive that the administrators are only motivated to create a culture of assessment for accreditation, cynicism and low-motivation will likely result” (Lakos and Phipps, 2004).

Next Steps


Formalize the Integration of Culture & Evaluation Model

Create an organizational evaluation audit to:Assess the stage of the model an organization is at

Develop strategies to assist moving it forward

Create a sustainability approach

Design and deliver the next phase of the research

Next Steps

Q: Do the agencies that fund your teen programs require that you report outcome/impact measures?


Ontario-wide survey to elected municipal officials focused on understanding what information they consider when in their deliberations on financial support to libraries’ budgets.

In part this research will provide insight for libraries in how to structure, and communication the results of, their institutional evaluations.

Defining New Metrics For Libraries Success iSchool, University of Toronto Symposium April 23, 2014...

Documents

Transcript of Defining New Metrics For Libraries Success iSchool, University of Toronto Symposium April 23, 2014...