7/31/2019 10 Workshop 5

1/38

Key Performance

Indicators: Adapting an accountability tool for digitallibraries

Leslie Wolf & Lena Zentall, California Digital LibraryDLF Conference, November 3, 2010

7/31/2019 10 Workshop 5

2/38

Agenda

Why KPIs Matter

Part 1: Brief Theory

Part 2: Practical Application (+Individual Exercise)

Part 3: Your KPI Action Plan(Individual Exercise)

Part 4: LookingForward/CommunityEngagement (Discussion)

7/31/2019 10 Workshop 5

3/38

Whats The Benefit Of Learning About KPIs?

You: Leader & Innovator

Your Organization: Track & Act

7/31/2019 10 Workshop 5

4/38

Why KPIs Are Valuable

What gets measured gets done

Anything you produce of value can

be measured

Create dialog and sharedunderstanding

7/31/2019 10 Workshop 5

5/38

Why Did Our Team Decide ToUse KPIs?

Incoming University Presidentlaunched new type of accountabilityinitiative

Decision-making tool

Tells our story

7/31/2019 10 Workshop 5

6/38

By The End of This Workshop

Youll be able to explain KPIs and developthem for your own institutions

Youll start an action plan

Youll have some handouts for reference

Youll understand how other participants areusing KPIs

Youll have an opportunity to get involved ina larger community of practice

7/31/2019 10 Workshop 5

7/38

Part 1: Brief Theory

What is a Key Performance Indicator?

What is the difference between statistics,metrics, and KPIs?

What are the components of a KPI?

What does a KPI look like?

Who uses them? Where do they find them?

7/31/2019 10 Workshop 5

8/38

Body Temperature as a KeyPerformance Indicator

Widely recognized andunderstood

Measured the same way byeveryone

Easy to collect and record

Critical indicator of health

Helps with diagnosis:If fever, investigate whyIf no fever, investigate differentsymptoms

7/31/2019 10 Workshop 5

9/38

More Definitions of KPIs

Easily collected from a reliable source

Quantifiable (can be calculated)

Reflects critical success measures

Measures progress toward goals

One of many tools in the management toolkit

A snapshot over time

7/31/2019 10 Workshop 5

10/38

Statistics? Metrics? KPIs?

Statistics for dailytracking. Some statisticsmay become

Metrics to manage your service. A very fewmetrics may become

KPIs for strategicplanning.

7/31/2019 10 Workshop 5

11/38

Possible Components for aKPI

Volumetransactionspage viewssearches

Cost

Timeliness

Quality

Customer satisfaction

7/31/2019 10 Workshop 5

12/38

How to Express a KPI

Sample Equation

Total Costs Number of Searches = CostPer Search

Two Ways to Express a KPI

1. Number of searches is aweak indicator

Is that a lot? Too little?

Whats the context?2. Cost per search is a better

indicator Ratio of 2 elements that arestrongly related

Provides more contextBetter diagnostic tool moreinformation to work with

7/31/2019 10 Workshop 5

13/38

KPI Example: Cost per Search

Quarter KPI

January February March $0.06 per search

April May June $0.09 per search

July August September $0.11 per search

7/31/2019 10 Workshop 5

14/38

Diagnosing Cost per Search

Quarter KPI Diagnostic

January February March

$0.06 per search

$15,000 cost/250,000searches

April May June $0.09 per search

$15,000/175,000Cost is stable butsearches are down

July August September

$0.11 per search

$20,000/175,000Cost is up and searchesare the same level as

April

7/31/2019 10 Workshop 5

15/38

Digging Down on Cost Per Search

Why are searches down?Users dont like the service or The recent upgrade made searches moreefficient

Why are costs up?Did a big upgrade and used moredevelopers

Can we release the developers andreduce cost?

Can we renegotiate the license fee?

What else can we do to change theequation?

7/31/2019 10 Workshop 5

16/38

Who is the Audience for KPIs

Traditionally for senior management

Typically reported via a dashboard

Goal is that in a few minutes a senior manager can get a quick update onimportant services

A senior manager can self-serve: no lastminute request for data

7/31/2019 10 Workshop 5

17/38

Simple Dashboard(Spreadsheet)

7/31/2019 10 Workshop 5

18/38

Cool Dashboard: IndianapolisMuseum of Art

7/31/2019 10 Workshop 5

19/38

Cool Dashboard: IndianapolisMuseum of Art

7/31/2019 10 Workshop 5

20/38

Part 2: Lets Get Practical

Well show you how we created KPIs for UC-eLinks, which is our name for SFX (ExLibriss link resolver software)

In this part, well let you brainstorm a fewpossible KPIs for your own service.

7/31/2019 10 Workshop 5

21/38

Step 1. Prepare

Determine who should be part of thediscussion.

Have a good mix of technical, productmanagement, and customer service.Invite senior managers to join in youllget quicker buy-in and approval thatway.

Send some background information aboutKPIs in advance (since its new to mostpeople).Determine what a successful meeting willaccomplish: what is the desired outcome?

7/31/2019 10 Workshop 5

22/38

Step 2. Brainstorm AllPossible Things to Track

What data do we already track regularly?In an ideal world, what would we want totrack?

What will senior management beinterested in?What do we care about most ?What KPIs will tell our story best?

7/31/2019 10 Workshop 5

23/38

Step 3. Make Sure You CanCapture the Data

After brainstorming, start to narrow downthe list to whats realistic.

If the data is too difficult to extract, your

effort will just fall apart.

Make sure your data source is valid andyou understand the definitions of the fieldsbeing captured.

7/31/2019 10 Workshop 5

24/38

Step 4. Narrow it Down to aFew KPIs

What we chose for UC-eLinks:Cost per Transaction (when a UC-eLinkswindow appears)

% of Successful Transactions (click- thrus)User Flow

Track where users are going using UC-eLinks(targets)

Track where users are coming from using UC-eLinks (sources)

Quality of Systems

7/31/2019 10 Workshop 5

25/38

Step 5. Define the DataElements

Define whats in a transaction.

The term means something different in eachdiscipline.

Define what costs you are going to track.What can you track without too much burden?

Define what you mean by quality of systems.It can be a mind- twister; dont make it toocomplicated.

Figure out ways to track user satisfaction . Are there existing surveys or trouble ticketingsystems?

7/31/2019 10 Workshop 5

26/38

Step 6. Put It In Writing

For cost: Well use staffing, infrastructureand licensing costs. We wont use costs for the 10 campuses.

For quality of systems: Well use a weightedaverage that our developers created. Welltrack Reliability, Availability, Maintainability,and Performance (RAMP).

For user satisfaction: Well use only reported

complaints since we have no survey data.

7/31/2019 10 Workshop 5

27/38

Now Its Your Turn (Exercise)

Interactive Discussion:

In your services, what do you think isimportant to track? For example:

Cost of serviceUser satisfactionQualityVolumeUser satisfaction

Would it be difficult or easy to track?

What could stop you? What are thechallenges?

7/31/2019 10 Workshop 5

28/38

Recap

We defined what a KPI is

We understand the difference betweenstatistics, metrics and KPIs

We know the components of a KPI and whata KPI equation looks like

Weve seen a reporting dashboard

We know how senior management usesKPIs

7/31/2019 10 Workshop 5

29/38

How Do You Get Started?

Get management approval for proposedKPIs

Set up a place to track the data (e.g. Excel

worksheet)

Get agreement on how often to track (wedecided quarterly)

Set up a test period (hands-off!)

Start tracking and reporting

At the end of the test period, assess andmodify as needed

7/31/2019 10 Workshop 5

30/38

What We Learned

Our timing was good. We had the data whenit was requested (no fire drills).

When everyone is focused on the KPIs the

team is more results- oriented: What getsmeasured gets done.

We started real dialog and sometimeslively discussions about how well weredoing (especially on quality of systems,

which is somewhat subjective).

KPIs are now integral to our serviceroadmap and strategic planning.

7/31/2019 10 Workshop 5

31/38

More Lessons

We hadnt accounted for all our expenses for instance, we didnt add benefits to our staffing costs . (We decided not to go backand change the test KPIs.)

Biggest problem: No benchmarks/baseline.We didnt know what was a good cost per transaction.

Whats the cost of improvement?

Whats the sweet spot to balance cost andquality?When is it good enough (whats theopportunity cost)?

7/31/2019 10 Workshop 5

32/38

Words of Wisdom for You

Everything that can be counted does notnecessarily count; everything that countscannot necessarily be counted.

Albert Einstein

7/31/2019 10 Workshop 5

33/38

More Words of Wisdom

You cant do this from the bottom up youneed senior management support.

Numbers tell a story understand that story

before you share it , not after you are askedto explain the results.

You cant control what senior managementdoes with the numbers when you leave theroom!

KPIs will reveal hidden truths.We worried: What if the service is moreexpensive and less effective than wethought?

7/31/2019 10 Workshop 5

34/38

Questions?

7/31/2019 10 Workshop 5

35/38

Part 3: Your KPI ActionPlan (Exercise)

Take a few minutes to fill out your action plan (see the Handout)

7/31/2019 10 Workshop 5

36/38

Action Plan Questions

Whats the first thing youll do when you getback to your office?

What was the most valuable thing you

learned today?

Key Points/Things you want to explorefurther?

What barriers, if any, do you foresee?

7/31/2019 10 Workshop 5

37/38

Part 4: Looking Forward

Interactive Discussion:

Where do we go from here?

Is there value in having a community of practice to share KPIs with peer institutions? If so,

Whats the value to you?

How do we make it happen?

7/31/2019 10 Workshop 5

38/38

Thank You!

Leslie Wolf leslie.wolf@ucop.eduLena Zentall lena.zentall@ucop.edu