Statistics for Improving the Efficiency of Public Administration
Daniel Peña Universidad Carlos III Madrid,
Spain
NTTS 2009 Brussels
Outline
1. Introduction 2. Building a quality index: attributes
and weights3. Methods to determine weights 4. A simple model 5. Extensions of the model 6. Conclusions
1. Introduction Public Adminstration provides
services to people (Health, Education, Transportation, Energy...)
Analysing the efficiency of these services requires measuring their cost and their Quality.
This talk is related to evaluating service quality
In order to evaluate and monitor the quality of a service we need to build an index (scalar measure) to summarize its performance: a quality index.
This a similar procedure to summarizing the evolution of the prices in a cost of living index.
Understanding the structure and the level of a quality index is needed for decision making about resource allocation and cost/benefit analysis.
We usually know how the budget is split among different projects or areas in an institution.
We do not know how different activities contributes to the overall quality of an institution.
Note that for services provided in a free market prices can be considered as a summary of the sevice quality.
This is not the case for public services where there is not an open market. For instance: basic research
We present a model for building a performance quality index which can be applied to the services provided for Public Administrations.
2. Building a Quality Index
It is assumed that the quality Q of a given service depends on some attributes:
Q=f(X1,…, Xk)For instance, for a university X1: quality of teaching = f(X12,...X1a ) X2: quality of research = f(X21,...X2b ) X3 : quality of innovation and transfer =
f(X31,...X3c )
University Quality
X1 (teaching) X2 (Research) X3 (innovation+Transfer)
X11 X21 X31 X1a X2b X3c
X111X11e
X1a1 X1afX311 X3ch
A linear quality Index (quality index from now on) combines all the different attributes into a single number:
Q = w1 X1 +…+wk Xk,
In order to build this index we need: The list of attributes. The weights.
The most important part are obtaining the weights because we can always write a longer than needed list of attributes and give to some of them weights equal to zero.
In most application of quality indexes (as for faculty evaluation) we do not have objective weights.
This is a great difference with a standard statistical index number, as Cost of living index, where the weights are objective (for instance the weight of a product in a cost of living index can be its relative contribution to the total cost of a familiar unit)
The procedure I will present can be applied to build a quality index for
A service process (student registration in a university or in a hospital)
An area of activity (research, teaching,..)
An Organization (A universtity or hospital)
3. Methods to determine the weights 1. For consensus among a small group
of experts. Advantage: it is simple to apply and can be presented
as a political decision Disadvantages: Different groups of experts can give
different weightsProblems of coherence with many
attributes
2. For indirect evaluation of the attributes and the quality from a sample of experts from some population of experts. The weights are obtained by statistical analysis.
Thus the data will be the evaluation of the overall quality of the service and also the levels of the attributes which determine the overall quality. For instance, overall university quality and also the quality of teaching, research and innovation
Two ways to carry out this evaluation: a) fix the values of the attributes
and ask for a global evaluation (value of Q). Then use experimental design linear models to estimate the weights.
This is conjoint analysis, we can use fractional factorials to evaluate many attributes
This procedure, which has been used for evaluating technical services (as telephone service), is not easy to apply when there is not and objective ways to fix the levels of the the attributes.
With three attributes each with two possible levels, eight possible services are defined and evaluated
evaluation
b) the second approach is to evaluate in several units both the attributes and the global performance ( o global quality) and use regression to build a model and estimate the weights.
Example: some universities are evaluated by defining the global quality and the quality of the attributes which determine the overall quality
The evaluation of some specific attributes is asked, X1=?, ….,Xk=? (for instance in a 0-10 scale)
The evaluation of the global performance Y is asked in the same 0-10 scale.
Then for each Judge we have the explanatory variables, X, the response o global performance, Y, and the regression coefficients will be the weights.
Problem: the weights will be different for different judges or referees and we want to estimate the distribution of weights in the population and the average weights to build the index.
4. A simple model
Note: This assumption is strong and can be relaxed as we will explain later
an example
5. Extensions of the model Instead of assuming the weigths
follow a normal distribution we can assume that they follow a Dirichlet distribution.
In the way: (1) they are positive and add up to
one; (2) the variability of each weight
depends on its importance;
We can test for missing attributes with the data allowing that the sum of the weights of the attributes considered to be smaller than one.
We can allow for mixture distributions for the weights (clusters of customers with different wishes)
We can predict the weights for each respondent and then relate these weights to personal characteristics to undertand the structure of people wishes
Further extensions
We can also assume that the attributes are evaluated with error, that is they are also latent variables.
Then we have a random coefficient model with errors in variables which is a challenging estimation problem.
We may also assume that we have measurement error in the X and Y and that both are multidimensional, that is instead of a response variable we have a vector of response corresponding to dimensions we do not want to put together
Then we may have a LISREL (linear structural relation model) model in which all the key variables are latent variables and are related to some observed variables, as:
The simple model we have presented have no measurement error in the X, have an scalar responde Y and assumes some joint distribution for the regression parameters.
LISREL models have been applied to Quality indicators usually assuming fixed, instead of random, coefficients in the relationship between Y and X.
An example: ACSI index
6. Conclusions
Building linear quality indicators seems to be an increasing important task for quality service evaluation
The key part of this task is estimating the weights
There are many important statistical problems link to this objective:
Cluster analysis to determine groups of customers with similiar weights
Factor and LISREL models and the EM algorithm for estimating these models
Multivariate Outlier Analysis for finding measurement errors and robust analysis
Multivariate time series analysis for monitoring these indexes over time
Non linear models for capturing the more realistic effects among the attributes and the overall quality
In summary, this is an important area in which statisticians can play a key role in the future.
Thank you for your attention
Top Related