Università di Bologna Dr. Luciana Sacchetti Dr. Ida D’Attoma University of Bologna Library of the...

Università di Bologna

Dr. Luciana SacchettiDr. Ida D’Attoma

University of BolognaLibrary of the Department of Statistics

Attention to the end user: how to measure quality in an ISO certified Library

7° Northumbria International Conference Performance Measurement in Libraries and Information Services Spier, South Africa 13-16 August 2007

The Library in 2006

Established in 1982 as a research Library in Statistics

32,000 collection

2,024 mt. of open shelves

6,39 FTE

Circulation 3082 (loans, DD, ILL)

Main users: Researchers of the Department of Statistics & students in Statistics and Economics

ISO 9001:2000 certified by DNV since 2004

Why adopting ISO 9001:2000

1. 360° management System for quality

2. A host of operational tools tailored specifically to the library’s profile

3. Attention to the end user

4. incremental and tangible improvements

5. The ISO “lesson learned” can be applied to different organizational context

The ISO QMS at work

FOCUS GROUPFOCUS GROUP

INTERNAL & EXTERNAL AUDIT


2.CUSTOMER SATISFACTION

SURVEY

2.CUSTOMER SATISFACTION

SURVEYPROCEDURES &

DESK INSTRUCTIONS

PROCEDURES &DESK INSTRUCTIONS

NONCONFORMITYCORRECTION


1.STATISTICAL INDICATORS

1.STATISTICAL INDICATORS

WORKING BY OBJECTIVES


QUALITY POLICYSTART/END


ISO 9001:2000QMS TOOLS


2.1 Cluster Analysis on

Principal Component

(work in progress)

Measuring/monitoring/evaluating quality

Working by measurable objectives

22 statistical indicators. Six of them monitor back-office procedures

Three others measure our user’s abilities and habits

The remaining are taken from the Customer Satisfaction Survey

The treatment of nonconformities consists of two steps:

time taken to process new material

the average usage of the physical journals & review section

1. monitoring and solving recurrent or minor nonconformities

2. discovering, analyzing and solving unpredictable or major ones

Measuring/monitoring/evaluating quality with QMS

The Customer Satisfaction Annual Survey finds out what the users know (or perhaps more importantly don’t know) about the Library, in terms of relevance, usage and satisfaction

Internal & External audit

The Focus group is annual qualitative survey to evaluate strategic topic with our main stakeholder group, the Professors and researchers in the Department of Statistics

10-15 minute interviews with students and junior researchers over a thirty day period.

The Questionnaire is divided in nine sections, the Satisfaction section being the most detailed

The INTERNAL - conformity to procedures and specifications

The EXTERNAL (DNV)-conformity of the System to the ISO

standard and the improvement achieved by the library

” how best to evaluate our current journal collections ?” (FC2004)

The library among Statisticians: an ideal context

The researchers of the Department of Statistics were very helpful during the implementing phase of the ISO QMS (statistical indicators, questionnaire set up etc.) Now that the System in fully implemented, we co-work to sharpen our measurement tools mainly focusing on the users expectations.

Here comes the statistical research made with Dr. Ida D’Attoma I will present you today.

The ISO QMS at work

FOCUS GROUPFOCUS GROUP



CUSTOMER SATISFACTION

SURVEY

CUSTOMER SATISFACTION

SURVEYPROCEDURES &

DESK INSTRUCTIONS

PROCEDURES &DESK INSTRUCTIONS



STATISTICAL INDICATORS

STATISTICAL INDICATORS







(work in progress) Cluster Analysis on

Principal Component

Descriptive vs Qualitative indicators

UNIVERSITY LIBRARY SYSTEM QUANTITATIVE INDICATOR

Loans total per year

Source: Data warehouseCalculus: centralized query

OUR OUTPUT2754 loans in 2006

OUR RANKING11° among 15 Social Science libraries of the University

QUALITATIVE INDICATOR OF THE LIBRARY

The “appeal” of the library’s books collection per year (%)

Source: QMS Calculus: ad hoc indicatorMethodology: total of borrowable books with at least one loan / total of borrowable books

OUTPUT1077/15,072 *100= 7% in 2006

BENCHMARKUniv. Library of Economics Sciences 9%Univ. Library of Management Sciences 14%

Comments &corrections

Loans total per year

From the University System point of view, the Library seems poorly efficient in meeting the information demands of the users – if compared to others with bigger collections and much more users & loans

The Image of the library is penalized

UNIVERSITY QUANTITATIVE INDICATOR

The “appeal” of the Library’s books collection per year

Since the average “collection’s appeal” is 10 % the Library 7% score seems good enough

Considering that:

50 % of our collection is not borrowable (others libraries >90 %)

the indicator is not fully efficient since it doesn’t show which kind or part of the collection is more “appealing” (new books, specific matters or disciplines…).

QUALITATIVE INDICATOR OF THE LIBRARY

Economics

THE CUSTOMER SATISFACTION SURVEY

The users’ segmentation by Faculty

Library’s users are mainly students from the Faculties of Statistics and Economics

0,69

17,93

1,38

0,69

1,38

0,69

0,69

6,21

70,34

B iology

E conom ic

Law

Inform at ic

Let ter

P hilosophy

M edic ine

P olit ical sc iences

S tat is t ic


The users' group

U s e r's g ro u p

72%

21%

6% 1%

student graduat ing s tudent P h.D S tudent Junior researcher

In 2006 we interviewed 145 individuals, with some sets of questions being targeted at specific user groups: students, PhDs, Junior researchers

Sample is mainly made up of students and so the analysis will only concern this users’ group

GLOBAL SATISFACTION FREQUENCY FREQUENCY%

1. Unsatisfied 0 0

2. Not much unsatisfied 0 0

3. Nor unsatisfied nor satisfied 0 0

4. Not much satisfied 6 5,71

5. satisfied 37 35,23

6. much satisfied 50 47,61

7. absolutely satisfied 12 11,42

A high percentage of users has a high level of satisfaction

The Customer satisfaction Survey


The users have attributed an average score greater than 4 to all services

In the measurement scale this corresponds to a higher importance, with not the same related frequency

The importance attributed to the services allows us to believe that exists a balance between the service’s offer and the user’s requirements

12

34 5 6 7 8 9 10

11 12 1314

1

2

3

45 6

78 9

1011 12

1314

0,00

0,50

1,00

1,50

2,00

2,50

3,00

3,50

4,00

4,50

5,00

Im portance Frequency


The average satisfaction scores toward general library’s aspects

Scores belong to high range of satisfaction

Satisfaction related on services

12

34

56

789

1011

1213

0,00 1,00 2,00 3,00 4,00 5,00

1

3

5

7

9

11

13

1 = Library’s opening Hours2 = cloakroom service convenience3 = reading room seating 4 = PC availability5.= reading room neatness 6 = reading room clinatization7 = reading room neatness in the basement 8 = lighting in the reading room9 = reading room clinatization in the basement 10 = clear information delivered11= respect for the library's regulations 12 = librarian’s professional competence13 = librarian’s helpfulness


Our users are mainly satisfied of:

•Bibliographic online databases•Web site•Wireless access

These aspects satisfy the demand of Library remote accessibility

1 = the books stock meets my requirements2 = copies of textbooks available3 = teachers notes availability4 = the borrow duration appropriateness5.= number of book that can be borrowed simultaneously

6 = OPAC7 = the Library’s web site8 = Wi-Fi Access9 = Photocopying facilities

4 ,0 5

3 ,3 13 ,7 3 3 ,7 4 3 ,9 6 4 ,1 7 4 ,1 8 4 ,4 1

4 ,1 4

1 2 3 4 5 6 7 8 9

S a tis fa c tio n re la te d o n ta n g ib le a s p e c ts

THE PRINCIPAL COMPONENT ANALYSIS Dr. Ida D’Attoma

We analyze user’s satisfaction questionnaire with cluster analysis on principal components obtained from a previous PCA procedure

The cluster analysis application to the Library is a “work in progress” aimed to go deeper into understanding the user satisfaction and his changing needs and requirements

Objectives are:

• to discover or to reduce the variables’ dimensionality of the data set • to identify new meaningful underlying variables• Eventually, to eliminate redundancy of some items of the

questionnaire

THE PRINCIPAL COMPONENT ANALYSIS

How it works

• PCA reduces the variables’ number that describe units (the users)

• Represents them by a smaller number of new uncorrelated variables called PRINCIPAL COMPONENT (or dimensions)

• PC - being uncorrelated by construction - are a linear combination of original variables: a weighted sum of the original variables

• So PC are unidentified variables that allow us to attribute new scores to units

PCA The reduction’s procedure

How many PC to retain?

We have started with 22 original variables of the Questionnaire

it is reasonable to retain 7 dimensions, because they accounted for 50% of data variability

(…) continue 14 to 22

Number Eigenvalue % explained

variance

% cumulated explained

variance

1 2,7684 12,58 12,58

2 2,4050 10,93 23,52

3 1,7900 8,14 31,65

4 1,6382 7,45 39,10

5 1,4432 6,56 45,66

6 1,2480 5,67 51,33

7 1,1422 5,19 56,52

8 1,0183 4,63 56,52

9 1,0183 4,55 65,70

10 0,9624 4,37 70,08

11 0,8841 4,02 74,10

12 0,8043 3,66 77,75

13 0,7782 3,54 81,29

PCA Examining the correlations

Once decided the number of component to retain, we assign a semantic name to each of them by examining the correlation between variables and components:

1. Efficiency of the lending service and professional competence of the Library staff

2. Efficiency of reading room and the web site

3. Library accessibility

4. Logistics & Infrastructures

5. Protection for the user’s property

6. Availability of textbooks

7. Quality of the book collection

Explained variance: 12,58%

Efficiency of the lending service and professional competence of the library staff

POSITIVE CORRELATION

Teacher's notes availability 0,40

The duration of borrow meets my needs 0,37

There are enough copies of textbooks 0,36

The number of book that can be borrowed simultaneously is sufficient

0,35

NEGATIVE CORRELATION

The Library Staff competence -0,52

The Respect for the Library's regulations -0,51

Clear information delivered at the Library's desk -0,46

Helpfulness of the library staff -0,46

PRINCIPAL COMPONENT 1

Explained variance: 10,93%

Efficiency of the Reading Room and the web site


The Reading room neatness 0,43

The Reading room climate 0,38

The Reading room lighting 0,33

The Reading room seating 0,19


The Web site -0,64

PRINCIPAL COMPONENT 2

PRINCIPAL COMPONENT 3 and 4

Component 3Explained variance : 8,14%

Library accessibility


The Opening hours 0,53

The Reading room seating 0,28

The Wireless access 0,21


The Reading room climate -0,41

The PC availability -0,38

The Lighting in the Reference rooms in the basement

-0,31

Component 4Explained variance: 7,45%

Logistics & Infrastructures


The PC availability 0,50

Photocopying facilities 0,29

The Cloakroom service convenience

0,25


The Electronic journals -0,57

The Opening hours of the L. -0,51

PRINCIPAL COMPONENT 5,6 and 7


Protection for the user's property


Cloakroom service convenience

0,59


Helpfulness of the Library Staff

-0,23

Respect for the Library's regulations

-0,19

Component 6Explained variance: 5,67%Availability of textbooks


There are enough copies of textbooks 0,45


Photocopying facilities -0,34

Teacher's notes availability -0,29

The duration of borrow meets my needs -0,27


Quality of the books’ collection


The book stock meets my requirement 0,27

Web site 0,25

The duration of borrow meets my needs 0,18


Electronic journals -0,27

There are enough copies of textbooks -0,23

PCA & REGRESSION ANALYSIS PC/global satisfaction

Based on PCA, we estimate the relation between each new PC and the Student’s GLOBAL SATISFACTION

To do this we have implemented a REGRESSION ANALYSIS

The PREDICTORS are the 7 component retained, and the DEPENDENT variable is the global satisfaction

Analysis of variance

Source DF Sum of squares

Mean square

F value Pr>F

Model 7 12.04487 1.72070 3.48 0.0023

Error 97 47.91703 0.49399

Corrected Total

104 59.96190

Root MSE 0.70284 R-Square 0.2009

Dependent mean 5.64762 Adj R-Square 0.1432

Coeff var 12.44497

Despite the low R-square’s value the model satisfaction is on the whole significant

The predictors explain the global user’s satisfaction

PC/global satisfationREGRESSION ANALYSIS

Significative PC’s on global satisfaction

PARAMETER ESTIMATES

Variable DF Parameter estimate

Standard error

t Value Pr > | t |

Intercept 1 5.64762 0.06859 82.34 <.0001

PC 1 1 0.08146 0.04122 1.98 0.0510

PC 2 1 0.09114 0.04423 2.06 0.0420

PC 3 1 -0.09043 0.05127 -1.76 0.0809

PC 4 1 0.08906 0.05359 1.66 0.0997

PC 5 1 0.09733 0.05710 1.70 0.0915

PC 6 1 0.02906 0.06140 0.47 0.6371

PC 7 1 0.17255 0.06418 2.69 0.0084

Only three components are statistical significant & influence the global satisfaction:

PC1 Efficiency of lending service and professional competence of the library staff PC2 Efficiency of reading room and web site PC7 Quality of Library’s books collection MANAGERIAL IMPLICATIONSMANAGERIAL IMPLICATIONS

FOR QUALITY IMPROVEMENTFOR QUALITY IMPROVEMENT

IMPO

RTAN

T!

Classification hierarchique directe

1 53 52 146 133 37 132 88 92 160 153 107 155 122 138 123 24 156 2 150 140 54 157 137 20 144 60 148 16 120 151 115 143 149 158 154 135 134 63 147 145 131 159 145 141 118 103 32 33 15

20.00%

41.90%

38.10%

CLUSTER ANALYSIS ON PC

Finally we implemented a cluster analysis procedure on the principal components obtained.

The result is groupings of like observations that are different from other groupings.

The choice of how many cluster retain depends of analysis purpose, managerial goal, budget constrain.

Our goal is to generate a cluster solution that results in a number of subgroups that generates useful information for the Library management, and with cluster characteristics easy to describe.

CA DENDOGRAM

CLUSTER ANALYSIS ON PC

Once the specific solution set is selected, we also describe the cluster obtained in terms of other variables that did not directly participate to the clustering procedure.

We asked what are variables that characterize the clusters.

We asked if the frequency or the mean of variables considered is higher or lower in the cluster than in overall sample.

CLUSTER 1 “THE DO IT YOURSELF”

Users are not from the Faculty of Economics, they live off the City Campus, go to Library 3 times a month.

They think our book stock doesn't meet their needs, rating this service as less satisfying.

They are less satisfied than the overall sample of our web site, neatness of lecture rooms, helpfulness and professional competence of the Library staff, respect for the Library's regulations, information delivered at the Information desk.

But they are most satisfied than the overall sample of teacher's note availability, reading room seating, PC availability, journals of any support.

They rate physical & electronic journals and internet access as the most important services.

Photocopies is the most used service, the reading room the less used.

They rate global service quality less than overall sample's rate

41,90% of the overall sample

Cluster 2 “THE CRAMMER”

Users are mainly from the Faculty of Economics, they don't live in the City Campus, describe Library as a place where they can study, not as a place with Pcs or internet facilities.

They think our book stock meet their needs, rating this service as most satisfying.

They are more satisfied than the overall sample of: lending service, number of copies of textbooks, the OPAC & web site, number of borrowing books, helpfulness and professional competence, respect for the library' s regulation, information delivered at the Info desk.

They are less satisfied than the overall sample of: teacher's notes availability, cloakroom service convenience, library's opening hours, reading room seating, lighting & climate, pc availability & wireless access.

They rate physical & electronic journals, library's staff assistance as the less important services.

Otherwise, they rate ”the open shelf” as the more important .

Lending and Italian Statistical sources are the less used.


Cluster 3 “THE GREGARIOUS”

They are not Economics students, but from Political Science.

They go to the library once a week, they describe it as a meeting place, not as a place where they can borrow books and other materials.

They are more satisfied than overall sample of: reading room climate neatness, lightning and seating, cloakroom service convenience, helpfulness and professional competence, respect for the library' s regulations, information delivered at the Information desk.

They are less satisfied than overall sample of: lending service, number of copies of textbooks, number of borrowing books.

They rate availability of lecture rooms and library's staff assistance as the most important services.

Photocopies is the less used service, and both teacher's note service and the Italian Statistical sources are the most used.


FINAL REMARKS (the Statistician)

ADVANTAGES

Applying the Cluster Analysis to identify underlying groups of observations within the data, allows us to account for user’s heterogeneity

We can answer the questions like:

“For what kind of people does quality system better work?”

“ Who are our the students attending the library?”

DISADVANTAGES

An important disadvantage of the proposed analysis stems from the clustering procedure

That is, the clustering procedure can result in many plausible solutions, with no apparent justification for choosing one over another

FINAL REMARKS (the Librarian)

ADVANTAGES

The research basically confirms our POLICY STATEMENT FOR QUALITY which main points are: staff competence & proactive attitude, collection growth according to the user’s needs, facility management

The “do it yourself” browser of electronic information, cohabits with a more traditional user - “the crammer” for who books are still valuable objects

DISADVANTAGES

CA on PC needs a high statistical competence to be applied

How much statistical competence is required to improve an ISO QMS?

How many Ida D’Attoma are available in libraries?

QUALITY ACTIONS CA on PC outcomes will be discussed at the library’s annual assessment meeting

The Questionnaire’s variables could be reduced

Grazie

Quality System Management [email protected]

Statistical procedures [email protected]

Video clips [email protected]

Images from the libraries of the University of Bologna Social Science Division and Library “Sala Borsa” Municipality of Bologna

Dipartimento di Scienze Statistichecopyright 2007

Università di Bologna Dr. Luciana Sacchetti Dr. Ida D’Attoma University of Bologna Library of the...

Documents

Transcript of Università di Bologna Dr. Luciana Sacchetti Dr. Ida D’Attoma University of Bologna Library of the...