Implementing a faculty assessment system: a...

Implementing a faculty assessment system: a case study at the University of Pittsburg, U S A

This monograph is part of the Institute's research on 'Improving the managerial effectiveness of higher education institutions',

directed by Bikas C. Sanyal, HEP

Implementing a faculty assessment system: a case study at the

University of Pittsburgh USA

by

John C. Weidman Professor

Education and Sociology University of Pittsburgh

Daniel С May Administrative Associate

School of Education University of Pittsburg

Paris 1994

U N E S C O : International Institute for Educational Planning

The views and opinions expressed in this booklet are those of the author and do not necessarily represent the views of U N E S C O or of the H E P . The designations employed and the presentation of material throughout this paper do not imply the expression of any opinion whatsoever on the part of U N E S C O or IffiP concerning the legal status of any country, territory, city or area or its authorities, or concerning its frontiers or boundaries.

The publication costs of this booklet have been covered through a grant-in-aid offered by U N E S C O and by voluntary contributions made by several M e m b e r States of U N E S C O , the list of which will be found at the end of the report

This volume has been typeset using IIEFs computer facilities and has been printed in IIEFs printshop

International Institute for Educational Planning (established by UNESCO)

7 - 9 rue Eugène-Delacroix, 75116 Paris

© U N E S C O 1994 nEPShd

Preface

Over the last decade, as a result of financial stringency, combined with demands for expansion of enrolments and improved efficiency, higher educational institutions have been forced to reduce expenditure, seek n e w sources of funding and improve the utilization of existing resources. This has necessitated changes in the mechanisms, techniques and styles of institutional management. At the same time, higher education has had to cope with increased diversification and n e w types of students, including adult learners, so as to meet the changing needs of the labour market and foster closer links with industry as well as widen participation through the introduction of distance learning.

The implementation of innovation and change in institutional management, however, often faces obstacles and problems, including internal resistance, inadequate staffing or financial resources to m a k e the change effective, or insufficient time devoted to preparing and planning for change.

It is against this background that in 1990 the IIEP launched a research programme on 'Improving the effectiveness of higher educational institutions' whose purpose was to increase understanding of the process of planning, introducing and implementing management changes in higher education institutions, in order to improve utilization of resources. The project aimed at identifying factors associated with success or failure, exploring ways of overcoming obstacles or problems, and suggesting methods to improve institutional management and increase the responsiveness of higher education institutions to changing financial, economic and social pressures.

The research programme has several components, i.e. an information base, case-studies and training materials and workshops. The case-studies were a particularly important element since they were designed to identify the factors and strategies associated with successful innovation and change, and show the obstacles and problems to be overcome. This

v

Preface

information was then subsequently used for the training materials and as a major input to the synthesis of the research programme.

Several types of innovation and change were pinpointed for particular study:

i) Change in the organization of institutions:

• N e w forms of decision-making structures and information flows.

s The merger of separate institutions, departments or units.

ii) Changes in financial management and resource allocation:

e Devolved budgeting. e Resource generation.

iii) Changes in educational delivery systems:

э From semester to trimester, from block to credit system, rationalization of curricula, double intakes.

iv) Changes in staff management, including staff development and appraisal.

In total, 14 case-studies and one desk-study were carried out, three each in Africa, Asia and Latin America and five in developed countries.

The study pubUshed in this volume falls under category (iv) above. This study has contributed to the understanding of the implementation of an assessment system of the faculty members of a university. The problems involved in introducing such a system in a well established university are extremely complex. In the present case, the problems were more acute in that the assessment was to be related to career promotion and salary increases in particular. The case demonstrates h o w these difficulties were encountered and h o w a consensus was reached in implementing the different stages of the system through tenacity and patience, extensive negotiations and revisions of the procedures over a period of six years.

vi

Preface

The case also shows the benefits the system has provided, e.g.,

i) in the rationalisation of the salary structure (enough funds were released from lower ratings of some highly paid faculty allowing the Dean to m a k e equity adjustments);

ii) in the increase of scholarly activities including increase in the volume of published work; and

iii) in achieving higher satisfaction a m o n g students in the classroom.

The present case provides a good example to those universities intent on improving faculty performance through the delicate and complex task of establishing an assessment system.

The overall results of the research programme will be published shortly in a synthesis of wide-ranging scope which covers the most important domains of university management

Jacques Hallak Director

vn

Contents

Preface

Introduction

Chapter I

Chapter П

Chapter Ш

Chapter IV

Chapter V

Chapter VI

Chapter V U

Chapter VIH

Appendix A

Appendix В

References

Background

The problem

Proposed solution

Planning the change

Implementation and problems encountered

Effects

Evaluation of the faculty assessment system

Conclusions and discussion

V

1

2

8

10

12

15

18

20

28

33

40

44

IX

Implementing a faculty assessment system: a case study at the University of

Pittsburgh, U S A

by John C. Weidman Daniel C. May

Introduction

A recent UNESCO-sponsored conference identified 'faculty evaluation, assessment, and accountability' (Weidman, et al., 1991) as a major area in which research was needed for higher education in developing countries. In an effort to address some of the major issues raised about faculty assessment during the conference, this paper presents a case study of the design and implementation of an annual faculty review process that was developed in conjunction with a merit-based system for determining annual salary increases. It focuses on the academic department (Administrative and Policy Studies) of which the senior author is chair, housed in the School of Education at the University of Pittsburgh. The case study outlines (a) the institutional conditions (i.e. budget reduction resulting from an institutional planning process) which led to the establishment of an annual faculty assessment system, (b) the design and implementation of the system, and (c) the results from an evaluation of its effectiveness. W e recognize that this particular university does not resemble those in most developing countries. W e believe, however, that m a n y of the underlying issues and concerns with respect to faculty evaluation have broader applicability to higher education institutions in other national and regional contexts.

1

Chapter I Background

1. Setting

The University of Pittsburgh is a research university (Carnegie Type I) offering a broad range of undergraduate and graduate degree programmes in the arts and sciences. It also has professional schools of medicine, dentistry, law, business, engineering, education, public affairs, social work, and library science. Several of its academic programmes and departments are among the best in the world. The main campus of the university, located in the Oakland neighborhood within the city of Pittsburgh, enrols 18,7Wfiill-time( 13,500undergraduate; 5,200 graduate) and 9,400 part-time (5,000 undergraduate; 4,400 graduate) students. The university receives public funds from the state of Pennsylvania amounting to about 20 per cent of its total operating budget.

The University of Pittsburgh's School of Education offers programmes for the certification of public school teachers, administrators, and supervisors of curriculum and instruction as well as masters and doctoral programmes in a variety of educational specialties. Most certification programmes are at the post-baccalaureate level with about 100 tenure stream faculty serving approximately 1,800 graduate students, 1,300 of w h o m are part-time.

This case study focuses on the Department of Administrative and Policy Studies. The department's mission is to prepare educational leaders, advance professional knowledge, and improve the research and practice of educational administrators, policy-makers, policy analysts, planners, and evaluators. Administrative studies contribute to the development of knowledge, skills, and values essential to the effective and enlightened leadership, administration and management of educational organizations in regional, national, and international contexts. Policy studies focus on the assumptions, decisions, plans, courses of action, and

2

Background

outcomes that occur as administrators, policy-makers, other educators, and citizens seek to achieve educational objectives, and include understanding the complex interaction between educational institutions and their cultural, political, and economic contexts.

Twenty-seven tenure stream faculty members currently serve 470 students. The department has only graduate degree offerings ( M . A . , M . E d . , E d . D . , and P h . D . ) in three programme areas: Social and Comparative Analysis of Education (Social Foundations, International and Development Education); Administrative Studies (Elementary, Secondary, and Higher Education); and Policy, Planning, and Evaluation of Education. Almost 40 per cent of our students are concentrating in the administration of elementary and secondary schools, and another quarter are focusing their studies on higher education administration. Close to 60 per cent of the department's students are pursuing doctoral degrees. The department has an excellent track record of preparing minority administrators for both basic and higher education, with approximately 10 per cent of our students coming from minority groups (virtually all African-American). Another 10 per cent of our students are from foreign countries. About 55 per cent of our graduate students are w o m e n .

Academic programmes in the department have been designed to integrate study across all areas represented in the department. This integration is reflected in the three core courses (educational administration and policy; social and educational theory; and disciplined inquiry) that are required of all doctoral students. There is also a c o m m o n written preliminary examination. In addition to masters and doctoral study in each of the three programme areas, Pennsylvania school administrative and supervisory certification are offered. Particular strengths of the department are its emphases on (a) the comparative and international dimensions of educational policy analysis, formulation, and implementation, and (b) the urban and multicultural environments forming the fundamental contexts for the educational enterprise.

A high level of integration across programme areas has been accomplished, with most faculty participating in more than one academic programme. Several faculty are engaged actively in research being published in some of the most respected journals in their respective fields. Faculty members have also held leadership positions in prominent professional associations. Three former presidents (Don A d a m s , M a r k Ginsburg, Rolland Paulston) of the Comparative and International

3

Implementing a faculty assessment system: a case study at the University of Pittsburgh

Education Society are members of the department's faculty as is William Cooley, a past president of the American Educational Research Association.

In 1986, the Danforth Foundation selected the department as part of the first cohort of four universities in the U S A to participate in a programme designed to support faculty w h o were developing model approaches for preparing educational administrators. The department used the Danforth grant to facilitate implementing the organization of the newly formed department, and supported subsequent work that included serving as a pilot site for the American Association of School Administrators' National Executive Development Centre. Local funding from the Frick Education Commission, Vira Heinz Endowment , Buhl Foundation, and the Calihan Foundation as well as a new grant from the Danforth Foundation support current work on an innovative programme for the preparation of school superintendents that is n o w being implemented with the assistance of Richard Wallace, the nationally respected (winner of the 1990 M c G r a w Prize for 'distinguished contributions to the advancement of education') former Superintendent of the Pittsburgh Public Schools, w h o will be joining the faculty on a full-time basis in 1993.

In 1990, William Bickel received a US$1.1 million grant from the Lilly Endowment for a joint project with the University of Pittsburgh's Learning Research and Development Centre ( L R D C ) to provide technical assistance for the evaluation of Lilly's educational programmes and for training minority evaluators. This award supports six doctoral students in the department's Policy, Planning, and Evaluation Programme. In 1991, I B M awarded the department a US$560,000 grant for a state-of-the-art computer lab with 20 I B M P C work stations linked to an A S 4 0 0 minicomputer that is used primarily for preparing students seeking to become educational administrators. In addition to providing instruction in the use of management software, this facility enables students and faculty to access the Pennsylvania Educational Policy Studies (PEPS) data base containing information on all 501 of Pennsylvania's public school districts that was developed by William Cooley and Maureen McClure.

Over the past two years, three new units have been incorporated into the department. In July 1991, the Western Pennsylvania Principals' Academy, funded by the Allegheny Conference on Community Development, became part of the department. This Academy provides continuing professional development to practising elementary and secondary school

4

Background

principals. In July 1992, the Tri-State Area School Study Council, a consortium of 100 regional school districts m o v e d into the department, as did the Institute for International Studies in Education, which houses a variety of international research and training projects.

2 . Organizational context

In 1985, a year-long institutional review was completed with the formulation of a five-year plan for the University of Pittsburgh. Underlying this plan were the basic assumptions that state funding for the university was likely to increase at a relatively low rate, if at all, and that there would be no significant increase in total student enrolment. Consequently, it was deemed necessary to develop a plan for internal reallocation of limited resources in order to maintain a high level of quality in already strong units as well as to enable investment in units with particularly strong promise of reaching excellence. The plan established academic, administrative, and budgetary priorities for the university. For the School of Education, the plan included a major re-organization and a 25 per cent budget reduction during the 1985/1990 period.

The re-organization of the academic and service components in the School of Education was to involve changing from a structure that had 13 relatively autonomous quasi-departmental units organized by narrow academic programme specialty (average size of 7 staff) to three departments ranging in size from 25 to more than 40. In addition, two School-wide service units were created to cover outreach to the local (Institute for Practice and Research in Education) and international (Institute for International Studies in Education) educational communities.

The Department of Administrative and Policy Studies was conceived in the 1985 plan for re-organization and formed in 1986 by bringing together all of the faculty w h o had been members of four, previously autonomous units: the Programmes in Educational Administration, Higher Education, Foundations of Education, and International and Development Education. A fifth unit, the Programme in Curriculum and Supervision, was divided, with two faculty members whose primary interests were related to administrative and policy concerns joining this department, in addition, two faculty members from the Programme in Educational Research Methodology whose major academic interests were in the area of evaluation moved their appointments to this department.

5


The primary means of budget reduction was to reduce faculty contracts from three (12 months) to two (8 months) academic terms per year. Faculty would have full-time instructional responsibilities during the semesters of peak student enrolment, A u t u m n and Spring. While the School of Education would continue to offer a variety of courses during the S u m m e r , such instruction was to be funded through a separate budget based on projected tuition revenue. This meant that faculty would be paid on a course by course basis during the S u m m e r period and that only those courses attracting enough students to at least 'break even' on cost would be offered. Such a system is quite c o m m o n in North American universities. Faculty w h o do not wish to teach during the S u m m e r are free to pursue other activities, including but not limited to research and consulting.

Faculty effort was to be reduced immediately following approval of the plan, but their salaries were not to be reduced accordingly. Rather, affected faculty were to have a salary freeze, with no increases for 2.5 years. In 1985, approximately two-thirds of the faculty were on three-term contracts and more than 90 per cent of the faculty were tenured. Since no tenured faculty were to be fired, there was a reduction in faculty full-time-equivalent (FTE) instructional effort (i.e. numbers of courses taught) without a corresponding reduction in the actual numbers of tenure stream faculty.

After university-wide faculty and administrative deliberation, The University Plan: 1985-1990 (which included the aforementioned reorganization and budget reduction for the School of Education) was approved by the Board of Trustees in March of 1985. While no tenured faculty members were to be terminated, an incentive programme was established to 'buy-out' contracts of tenured faculty w h o wished to leave the university, thereby reducing the size of the faculty. Ultimately, very few faculty members were able to take advantage of the 'buy-out' option within the short time (2 years) that it was available. Consequently, the primary means of budget reduction was through the 2.5 year salary freeze associated with the mandatory shift from a three-term to a two-term annual faculty appointment. The total overall budget reduction was also decreased by the need to fund essential instruction during the third term (Summer) when m a n y public school personnel wanted to take courses at the university.

6

Background

A n e w Dean of the School of Education arrived in February 1986 to oversee the implementation of the massive re-organization. In order to infuse a somewhat entrenched faculty with fresh energy and in anticipation of contract 'buy-outs' as well as normal faculty retirements, the n e w dean negotiated an agreement with the university administration to allow the appointment of up to 25 n e w faculty members over a five-year period. Even with the newly hired faculty members, a 25 per cent reduction in the actual number of tenure stream faculty was anticipated.

The three n e w departments in the School of Education were formed officially on 1 July 1986 and required to develop plans for the structure of their academic programmes. These plans had to be approved by School of Education as well as appropriate University graduate bodies. Internal governance structures for the departments also had to be established.

The Department of Administrative and Policy Studies was formed with 28 tenure stream faculty members, including all of the faculty from four (Educational Administration, Higher Education, International and Development Education, Foundations of Education) and individual faculty from three other (Educational Research Methodology, Curriculum and Supervision, Counselor Education) of the former 13 programme areas. Only two of the faculty were untenured assistant professors, and the remaining tenured faculty were about equally divided between associate and full professors. In addition to the faculty, the department was staffed with four full-time secretaries and a full-time administrative assistant.

7

Chapter II The problem

The immediate problem was to satisfy the mandate from the university's chancellor that all annual salary increases be awarded on a merit basis, which required that an annual faculty review process be established. A closely related objective was to develop procedures that would improve the productivity of an entrenched faculty and staff. It was hoped that this problem could be addressed by changing the reward structure so that productivity was reflected directly in annual salary increases. For faculty, this meant basing salary increments on evidence of accomplishments in the areas of research, teaching, professional activity, and service. It was assumed that faculty would be motivated to improve their performance across the four areas if they knew that their annual salary increases would be related directly to the results of their annual reviews. A further assumption was that the annual faculty review process could be used as a basis for faculty development, since any weaknesses in performance could be identified in the annual review and steps taken to provide opportunities for improvement.

1. Establishing a system for documentation and evaluation of

faculty performance

If there was to be an application of a merit-based system of awarding annual salary increases in each of the areas of research, teaching, and service, a consensus would have to be developed on what sorts of evidence would be necessary to make a judgment in each area of faculty effort. Furthermore, some criteria would have to be established in order to make judgments about the quality of individual faculty members ' performance. A scheme for weighing evidence and assessing its actual contribution to the over-all annual faculty review procedure was also desirable.

8

The problem

2 . Improving faculty salary equity

It was assumed that salary inequities among faculty could be reduced if judgments were based on clear and consistent criteria over a period of time. This would occur because unallocated funds from the salary pool due to smaller than average raises for faculty getting low ratings could be re-allocated to increased raises for faculty with high annual review ratings.

9

Chapter III Proposed solution

1. S y s t e m for annual faculty review

The system for annual faculty review was originally proposed in 1986 by the newly appointed Dean of the School of Education in response to the mandate by the university's central administration that all faculty salary increases be allocated on a merit basis. The areas of faculty performance on which judgments were to be made were taken from the description of expectations for faculty work published in the University of Pittsburgh Faculty Handbook. The area of 'Professional Activity' was added to the conventional dimensions of 'Teaching', 'Research', and 'Service'. This was done to reflect the contributions made by faculty in a professional school to publications and professional groups that were not necessarily academic in nature, but which contributed materially to the educational enterprise. The new Dean also was adamant that, because faculty were working in the environment of a research university, there should be a particular emphasis on rewarding research productivity. Further, only those faculty with outstanding publication records could qualify for the highest merit rating.

2 , Objectives of the faculty evaluation system

These were finally established as follows:

s Provide consistent and fair assessment

Probably the most important objective was to develop a system that would ensure consistent and fair assessment of annual faculty performance across a diverse faculty, both within departments and across the School of Education.

10

Proposed solution

® Motivate faculty to increase productivity

It was hoped that the annual faculty review system would motivate faculty to increase productivity in all areas of effort (research, teaching, professional activity, and service) because salary increases were to be tied directly to level of performance across the four areas. Further, the special weight given to research would not only increase faculty research activity but also the increases in research productivity would enhance the standing of the School of Education in national and international higher education communities.

• Reward highly productive faculty

Highly productive faculty would be rewarded for their efforts on a yearly basis. In addition, equity adjustments in salaries would be m a d e where appropriate by sMfting funds for raises from the least to the most productive faculty.

ö Identify faculty development needs

Information from the annual review dossiers could be used to identify strengths and weaknesses of individual faculty members . Because the annual review process included the opportunity for faculty to meet with their departmental chairs and/or the School of Education deans, it would be easier for department chairs to identify faculty development needs.

11

Chapter IV Planning the change

The annual faculty review procedure was planned throughout 1985 but its refinement continued over a five-year period, from the Winter of 1986 through the Spring of 1991. K e y actors in the process included School of Education deans and department chairs as well as members of Faculty Council, the elected faculty governance body. The primary responsibility resided with the dean and department heads w h o conducted the annual faculty reviews.

It should be particularly mentioned here that in the area of staff evaluation, which by its nature is a delicate and complex operation, 'planning' cannot be complete. In the case of the university's experience, procedures as well as criteria for judgments across the four areas of faculty evolved on a consensual basis over a period of five or six years after the system was implemented.

However, various basic documents and procedures had to be prepared before implementation, as follows.

1. Guidelines

General guidelines for annual faculty review were developed by the Dean of the School of Education in consultation with the associate deans, institute directors, department chairs, and Faculty Council. A s stated above, the areas in which faculty performance was to be judged were based on the description of expectations for faculty performance but with a particular emphasis on research. The Dean also sent a document to faculty that explained in some detail (a) the kinds of information required for each of the four areas of accomplishment, (b) the nature of the review process, and (c) the dates when materials were to be submitted. This document receives minor refinements each year, but continues as the primary source of directions for the annual review process. Appendix A

12

Proposed solution

contains a copy of the document sent out to faculty prior to the 1991 annual review.

2o Procedures for submission

Procedures for submission of documentation were initially established by individual department chairs, but uniformly required the submission of a curriculum vitae containing the information requested by the guidelines with a personal statement from each faculty m e m b e r optional.

3» Merit categories

The Dean established a set of categories to be used for the merit assessment and detennination of associated salary increases: Unsatisfactory (no salary increase), Satisfactory, Merit, High Merit, and Exceptional Merit. In order to get a broader distribution of salary increases, all categories except Unsatisfactory and Exceptional Merit were to have three levels, resulting in an 11-point scale. The Dean mandated that research productivity be given the greatest weight in the review process. There is, however, continuing discussion of the relative weights that should be given to accomplishments in each of the four areas (teaching, research, professional activity, service), including qualitative assessments of evidence within each category.

4 . A n i m a l faculty review process

A three-stage review process was developed by the dean, in consultation with department chairs. In the first stage, department chairs read all dossiers and meet with any faculty members wishing to discuss their annual reviews. This part of the process culminates with the chair sending to the Dean a written recommendation for the level of merit achieved by each faculty member .

The deans then read all dossiers (including the chairs' letters of transmittal) and meet with individual department chairs to discuss within-department ratings. This provides the chair with the opportunity to make a case for the particular efforts of individual faculty members , especially

13


recognition of contributions in the areas of departmental governance and service.

In the final stage, the deans, department chairs, and chair of Faculty Council meet to discuss the dossiers and determine consensus ratings of all faculty in the School of Education. This is done in order to ensure that there is a high degree of consistency across the entire School of Education. The entire process is very time-consuming, with the largest effort required of deans and department chairs in reading and evaluating dossiers.

A n appeal procedure was established so that faculty members could dispute both departmental and final consensus evaluations. S o m e efforts have also been made at both the departmental and School of Education levels to provide opportunities for faculty development so that improvement could be made in problematic areas.

14

Chapter V Implementation and problems

encountered

W h e n the annual review system was introduced in 1986/1987, faculty resistance was very strong. Most favoured 'across-the-board' pay rises equal to the annual percentage salary increase allocated by the university's central administration. The majority tended to be unwilling to accept procedures forjudging differential performance, either their o w n or those of colleagues, nor were they particularly enthusiastic about the rather extensive and time consuming process of preparing a dossier.

There was strong resistance from a significant number of faculty to placing the greatest value on research productivity. These faculty felt that teaching, student advising, and service activities were being slighted. Further, some resentment was expressed toward faculty heavily engaged in research w h o were perceived to be contributing less than their fair share to departmental committee work, but w h o were, nonetheless, being highly rewarded. In the first two years following implementation of the annual faculty review process, one faculty m e m b e r in the Department of Administrative and Policy Studies refused to submit a dossier, one resisted submission until the latest possible m o m e n t , and several others c o m plained about the annual review process. Several faculty members believed that, rather than for developmental purposes, the review process was being used to deny raises, and thereby release funds for re-distribution to highly productive researchers.

Initial resistance was overcome, in large part, by inertia. Fully two-thirds of the faculty were not eligible for pay increases during the first two years of implementation due to previously mandated reductions in their contracts from three to two terms per year. Even in the third year of implementation (1988/1989), these faculty were only eligible for a m a x i m u m of half of the university's 6.5 percent salary increase. Hence, there was little motivation to be very serious about the annual review process until the following academic year (1989/1990), when all faculty

15


were eligible to compete for the full amount of the funds available for salary increases.

A n immediately perceived problem was that of comparability of materials submitted by departmental faculty to document accomplishments which led two of the chairs to develop a form to guide construction of departmental faculty dossiers for the 1987/1988 annual review. The submission of a curriculum vitae continued to be required of each faculty member . While this improved comparability of annual review dossiers within the Department of Administrative and Policy Studies, it did not solve the continuing problems with comparability across all three departments. Hence, in addition to the need for a detailed set of guidelines about what sorts of evidence would be required to document faculty accomplishments in each of the four areas, there was also a need for the dossiers to be structured so that the information provided was comparable across the entire School of Education.

The deans and department chairs, in consultation with Faculty Council, constructed a School-wide 'Suggested F o r m ' for use in the 1988/1989 annual faculty review. This form was refined and its use 'encouraged' for all subsequent annual faculty reviews. The form includes a request for submission of materials documenting performance (course syllabi, student evaluations of teaching, copies of presentations and publications, etc.) along with a curriculum vitae. Out of this a form has evolved for guiding the preparation of an annual review dossier that n o w tends to be used by virtually all of the faculty in the School of Education. This 'Suggested F o r m ' is included in Appendix B.

Throughout the three-year implementation period, there was faculty and administrative discussion of the procedures and processes which resulted in several modifications. S o m e faculty members also continued to resist (a) submitting extensive documentation of their accomplishments on an annual basis, (b) using suggested forms, and (c) being rated on a scale that was perceived to be less than precise. Faculty also continued to object to having research and scholarly productivity be the most important factor in the rating process. Department chairs complained that faculty w h o made significant service contributions to the department in areas such as committee work, but w h o did not report m u c h scholarly activity, were becoming disenchanted because of the lack of reward for service activities essential to the functioning of the department.

16

Implementation and problems encountered

Consequently, there was expanded consideration of contributions in the areas other than research after 1989.

17

Chapter VI Effects

1. Apparent achievement of Initial objectives

There seemed to be a reasonable consensus among the department chairs that (a) the annual review and rating system was identifying reasonably well those faculty whose performance warranted merit salary increases, and (b) the merit categories assigned as a result of the annual faculty review allowed for a clear and consistent assignment of merit pay increases. There seemed to be enough funds released from lower ratings of some highly paid faculty to enable more equitable allocation of the limited pool of funds for annual salary increases to lower paid, but more highly meritorious, faculty. This also seemed to allow the Dean sufficient budgetary slack to m a k e equity adjustments (a) for faculty with inordinately low salaries, and (b) for salary compression of junior faculty. B y the end of the fourth year of the system's use, two of the three department chairs felt that faculty had actually increased the level of their scholarly activity and were beginning to increase the volume of published work. There was also a sense that students were more pleased with the classroom dimension of the faculty's work (See Chapter 7, Evaluation, below).

2 . Continuing problems

There continued, however, to be a variety of issues raised by faculty about the merit rating process. S o m e faculty whose research productivity was very low, but w h o spent considerable effort in working with students and/or in departmental service activities, tended to receive lower than average salary increases, resulting in some disaffection.

18

Effects

S o m e highly rated and rewarded faculty were perceived to be contributing less than their fair share to student advising and departmental governance. This led to some resentment among the faculty of those colleagues w h o were perceived to be spending their time doing research at the expense of teaching, student advising, and departmental committee work.

The assessments of faculty review dossiers by deans continued to be somewhat lower than those by department chairs. A fundamental reason for this was related to the chairs' efforts to maintain good relationships with departmental colleagues. Hence, w h e n a colleague was contributing substantially to the department in terms of service (e.g., committee work, curriculum development), chairs tended to rate that person higher than did the deans. A s one chair said, "The Dean can be the bad guy. I ' m the one w h o has to deal with colleagues on a daily basis." Ultimately, the merit rating process was based on the consensus across all four areas (research, professional activity, teaching, and service) that evolved among the deans and department chairs. 'Objective' criteria for rating faculty in each of the four areas were never developed, despite requests each year from faculty to specify in some detail what they were.

19

Chapter VII Evaluation of the faculty

assessment system

That faculty were motivated to improve their overall performance is suggested in Table 1 which shows that the distribution of ratings shifted rather dramatically toward the upper categories in 1990, the first year in which all faculty were eligible for full merit raises allocated on the basis of the annual review process. There was a further upward trend for the annual reviews completed in the Spring 1991 and 1992. Both of these reviews were conducted under an interim dean, later appointed dean, w h o had previously participated in the process because he was one of the School of Education's two associate deans. In the first three years of the system's implementation, the Dean insisted that there be a 'bell-shaped' distribution of merit ratings, with the middle category of 'merit' being the most numerous. This type of a distribution is reflected in the data for the years 1987 through 1989 shown in Table 1.

The major focus for the following evaluation of the faculty assessment system is research and scholarly productivity, since that was the dimension mandated by the Dean as being of pre-eminent importance in determining the ratings. Not surprisingly, it is also the most readily quantifiable dimension. A s initially implemented, there was no special consideration given to age or stage of career (Centra, 1989). All staff were held accountable for research productivity based on the very same judgment criteria.

1. Research a n d scholarship

In order to gauge the impact of the annual review process on this aspect of faculty work, it was important to compare scholarly productivity in the years prior to the implementation of the assessment system with what faculty did in the years following implementation.

20

Evaluation of the fa

culty assessm

ent system

^ о

№ S

<3~ vo

С

e= ^

^ wn

Щ

о C3\ o\

,_ G

\ ON

<N

G\

ON

21


Since most faculty were not eligible for full salary increases during the first three post-implementation years, for the analysis the post-implementation phase was divided into two periods, 1987/1989 and 1990/1992.

The curriculum vitae submitted as part of the annual review dossier were used to obtain information about faculty scholarly activity during the twelve-year period from 1981 through 1992. Further, the twelve-year quantitative comparison was restricted to those tenure-stream faculty w h o had been employed continuously (including sabbatical and other leaves) at the University of Pittsburgh for the entire period (more than 80 per cent of the tenure-stream faculty). The three scholarly productivity indicators used were simple counts of the actual numbers of vitae listings by year of (a) presentations at professional meetings, (b) published books and book chapters, and (c) published journal articles. Simple arithmetic means were calculated across the time periods mentioned in order to obtain a m e a n annual rate of productivity in each of the three areas. Agreement among at least three different raters on these publication/presentation counts tended to be around 90 per cent. Because of the wide range of types of presentations and publications usted on faculty vitae, no effort was m a d e to weight the counts by some index of quality. The counting process corresponded generally to the approach taken in the consensus reviews conducted by the department chairs and deans.

The letters reporting the outcome of departmental faculty members ' annual review written by the dean and the chair were used to provide supporting information for the evaluation. In addition, information about faculty attitudes was obtained from letters, Faculty Council minutes, and other documents.

T o get a sense of h o w faculty in the Department of Administrative and Policy Studies stood in relationship to colleagues in the School of Education, means were calculated for all tenure-stream faculty in both categories w h o had been continuously appointed for the 1981/1992 period. Finally, scholarly productivity was divided according to the annual review rating category assigned to each faculty member . Tables 2 and 3 show the results of these analyses.

With respect to the School of Education faculty as a whole, Table 2 suggests that only for presentations were the post-implementation means larger than the pre-implementation mean .

22

Evalu

ation of th

e facu

lty assessmen

t system

£ МЭ

О

- ГА

ft

ОС Ж

ON g

ON С;.ON

£ S ^ §

ОС ОС

ON J

ON ON-

ON

5 5

О

ON CN

oo ON

ON ON

О ^

и

Z es"

с 'S о 23

Implem

enting a faculty assessment system

: a case study at the U

niversity of Pittsburgh

Q.

ГЗ -.r. 3 О

3 'О

SO

У r-с >•> Х2 У

:

о "В

> В

ГЗ С

Л

> Я

'•=

О

О

CU

3

"° "Я 2

а а

и j^

> i—

'¿

3 ГЗ

?3 —

1—

í¿

to

•з -а :л

.i—

> S

з л

v2 -S >> >

-. .—« "*—* *-" "^ ГЗ

Р о

у >

£

'С и S igh

ас

~ <

гп

Х> гз

0\

ON (N

OO

O

N

ON

ON

<

¿

2 p

q ^н

ON

oo O

N

f 00 ON

T—

1

<N

ON

O

N

о1

& •—

'

—с

cd H

О

Ü

NO

oo

O

N

J oo

O

N

I—'

ON О

^

о

CS O

N O

N

О ON

24

Evaluation of the faculty assessment system.

Table 2 also shows that each of the three types of scholarly productivity was directly related to the merit rating assigned. This suggests that the Dean's goal of making publication productivity the primary determinant of faculty merit rating was accomplished. Differences in scholarly productivity between the two post-implementation periods within merit rating categories suggest some relaxation of the initial emphasis on volume of productivity.

Table 3 shows that faculty scholarly productivity patterns in the Department of Administrative and Policy Studies were also directly related to merit ratings during the 1990/1992 period, but during 1987/1989 only for journal article and monograph productivity. This was not as consistent as the findings shown for the entire School of Education faculty because productivity of presentations and books for those receiving 'Merit' ratings was higher than for those receiving 'High Merit' ratings, but only during the 1987/1989 period. For the departmental faculty as a whole, all three types of scholarly productivity were greatest in the full salary incentive period, suggesting that the annual faculty review system did provide some motivation for increasing publication productivity. For departmental faculty, the most striking area of increased productivity was books and book chapters.

Given the sometimes inconsistent patterns shown in Tables 2 and 3, a 'turnover' analysis was conducted for journal publication to determine whether changes in productivity patterns were consistent across individual faculty members . Table 4 shows that 18 (56 per cent) of the 32 faculty in the entire School of Education w h o had not published journal articles in the pre-implementation period also published none afterwards. This represents only 22 per cent of the continuously appointed faculty in the School of Education. Another 11 faculty members w h o reported journal publications before 1987 did not publish any more journal articles after the annual review system was implemented.

A m o n g faculty in the Department of Administrative and Policy Studies, Table 5 shows that only 3 of 22 (13 per cent) did not publish in journals between 1981 and 1992. Further, 8 of the 11 (73 per cent) w h o had no journal publications prior to 1987 published a journal article afterwards.

25


Table 4 . Change in school of education faculty journal publication patterns (continuous appointees)

Average yearly journal articles, 1987/1992

None 1 or 2 3 or more (N)


None 18 10 4 (32)

1 or 2 9 4 3 (16)

3 or more 2 14 19 (35)

Table 5. Change in administrative and policy studies faculty journal publication patterns (continuous appointees)


None 1 or 2 3 or more (N)


6 2 (11)

2 1 (3)

4 3 (8)

It must be noted that those School of Education faculty appointed after 1986 were deliberately excluded from the analysis because they were hired into a situation in which the faculty performance expectations had already been deUneated through the annual review process. In fact, in the 1991 annual review, the group of 16 tenure-stream faculty appointed after 1987 had annual means on presentation and journal productivity that were

None 3

1 or 2 0

3 or more 1

26

Evaluation of the faculty assessment system

scholarly productivity were also unequivocally related to merit rating. Further, patterns were similar for n e w faculty in both the department and the School of Education. O f course, it must be noted that the n e w faculty were appointed largely on the basis of their potential for high scholarly productivity.

2o Teaching a n d service

The teaching and service dimensions of the annual review system were not analysed because there was really no specific set of indicators available. While faculty are encouraged to submit student evaluations of their teaching using a standardized, normed instrument administered by the University of Pittsburgh's Office of Measurement and Evaluation, m a n y do not. A number of faculty have argued that this machine- scorable student evaluation of teaching questionnaire is inappropriate for graduate courses because it was originally developed and normed for use in undergraduate courses. This issue is currently being addressed through the pilot test of a n e w , faculty-designed, student teaching evaluation form for School of Education courses. O f course, there are also those faculty w h o question whether a machine-scorable instrument can reflect adequately the nuances of graduate teaching and advocate collecting different types of data from students as well as using colleagues evaluation of teaching.

3 . Salary equity

N o analyses of the impact on salary equity were done. Because most faculty were eligible for full salary increases only in 1990 and 1991, not enough time had passed to expect any significant change in salary distributions that could legitimately be attributable to the faculty assessment system. Judging salary equity is extremely complex, in any event, due to differences in faculty ages and experience, including years in academic ranks. Further compounding the difficulties in this case are the large ranges between the highest and lowest faculty salaries (more than US$60,000) , and the longest and shortest tenure (more than 30 years).

27

Chapter VIII Conclusions and discussion

The annual faculty review system did accomplish the objective of providing a consistent and comparable basis against which to judge accomplishments. It also made publication productivity the primary determinant of a faculty member ' s merit rating. After six years of using the system, faculty members in the Department of Administrative and Policy Studies were submitting dossiers containing requested information on each of the areas of effort considered. Further, a strong consensus had developed among deans and department chairs about what evidence of accomplishment warranted classification into each of the merit categories.

Faculty continued, however, to find fault with the annual review system. Throughout the implementation of the annual review process, faculty complained that it was being driven by administrative concerns and was not sensitive to the complex demands on faculty from students as well as from educational personnel outside the university w h o frequently sought their advice. In short, a merit system loaded toward research and scholarship was not perceived by faculty as reflecting the scope and complexity of their actual work.

In response to these concerns, the Dean invited the School of Education's Faculty Council to address the annual review process and recommend modifications. H e also invited Faculty Council to suggest ways that the faculty might actually establish a structure (possibly similar to the promotion and tenure committee already elected by faculty) to do the annual faculty review independent of administrators. Despite yearly requests by the Dean for specific procedural suggestions, none were forthcoming from the faculty.

Probably the major shortcoming of the annual faculty assessment system was that few institutionally funded development opportunities were available to assist those faculty with performance shortcomings in one or more of the four areas. While there was some discussion of providing

28


support for faculty training in research design and methods, none was forthcoming except that either doing or learning to do research was m a d e a major criterion for being awarded a sabbatical leave. Hence, an obstacle to bringing about effective change in faculty performance was the lack of a systematic advanced professional development programme linked directly to the annual review process (Centra, 1989). In the absence of such opportunities, responsibility for motivating faculty to learn h o w to improve in areas of weak performance fell mainly on the department chairs, a finding that is consistent with other studies (Baird, 1986; Cresswell and Brown , 1992).

1. Improving faculty research performance

The results of the evaluation suggest that the annual faculty review system did not appear to increase long-term (those employed over the 1981/1991 period for which data were collected) faculty performance in the area of research and publications. Faculty did m a k e more professional presentations after the annual review was implemented, but this increased activity did not seem to lead to higher publication rates. It is highly likely that the increased funding provided by the n e w Dean after 1986 for attendance at professional meetings, available only to faculty w h o were making presentations on the programme, was a motivating factor behind their increased participation.

The new Dean also obtained grants from local foundations to support collaborative research projects by faculty. This funding was intended to support teams of faculty that included both productive and non-productive researchers, in the hopes that they would serve as a sort of professional development research apprenticeship. While no specific evaluation of these projects has been completed, the absence of overall increases in faculty research productivity during the period the grants were available suggests a lack of impact.

Looking only at the 1991 annual review, faculty hired after 1986 did exhibit higher publication productivity, as a group, than their longer-term colleagues. A single year's data are not, however, sufficient for any conclusions to be drawn about the prospects for the continued research productivity of newly hired faculty. Because of the very competitive academic labour market in the U S A , most of the newly hired faculty had established publication records prior to their appointment.

29


With respect to teaching and service, no comparable empirical indicators were available. Even though there is a university policy that faculty are responsible for making certain that each of the courses they teach is evaluated on a regular basis, faculty were free to choose both the type of teaching effectiveness evidence they submitted with their annual review dossiers and the particular course for which it was provided. M a n y , but certainly not all, faculty used the normed student evaluation of teaching form administered by the university. Service was documented by a listing of activities, but there was no way to determine the actual level or quality of contributions m a d e by an individual faculty member . A n assessment of annual review letters written from 1987 to 1991 suggested Utile change in either of these dimensions. The letters did identify concerns about the teaching or service activity of individual faculty members but, with a few exceptions, there was no consistent pattern to these instances.

2 . Planning a n d implementing change

Implementation in this case was very clear and direct. Faculty were provided with specific information about the annual review process, with descriptions of the types of evidence that was required to document performance, and with letters describing the outcome of the assessments m a d e by their o w n department chair as well as the final assessment by the dean. There was also an appeal procedure which included a second review of the faculty member ' s dossier. Over the 1986/1991 period, there was an average of two departmental appeals per year, about 25 per cent of which were successful.

Motivating faculty in an institution of higher education to change their behaviour is a very complex and difficult endeavour. Research (Bowers, 1989) suggests that there should be strong faculty involvement in and consensus about the design for any faculty evaluation system. C o m m e n t was solicited from Faculty Council on the descriptions of the types of effort required in each of the areas of assessment that were written by the Dean, but none was received. Further, in this particular institutional context, faculty are reluctant to judge one another for the purpose of awarding annual salary increases. This leaves the assessment in the hands of departmental and School of Education administrators rather than non-administrative colleagues.

30


3. Designing the evaluation

The fundamental problems in this type of evaluation are (a) identifying relevant evidence for each area of effort, (b) obtaining comparable evidence from all individuals involved, and (c) determining the relative importance of different types of evidence. O f course, it is also necessary to understand the basic objectives of the system which is being evaluated. Because this particular case is based on an academic department in a research-oriented university with extensive doctoral programmes, improving faculty research productivity was the driving objective. Gearly, other types of institutions with different missions in different national contexts would have to design faculty evaluation systems that were relevant to their o w n particular situations, with special attention to faculty rewards and institutional resources.

While not discussed here, there is m u c h relevant literature on issues related to teaching (Weimer and Lenze, 1991) and public service (Crosson, 1983) in higher education as well as on the interdependence of faculty teaching, research, and service activities (Fox, 1992) that could be used in the design, implementation, and evaluation of faculty assessment systems. While issues in the design raised by the present case study m a y be appHcable for universities in other countries, it ultimately is the responsibility of faculty and administration within local institutions to identify those dimensions that are most important for their o w n specific contexts.

Our experience with the annual faculty review and merit evaluation process in the School of Education at the University of Pittsburgh suggests that the incentive represented by merit-based salary raises is important but m a y not provide sufficient motivation for faculty to increase their research productivity. Even after the annual review and merit pay system had been implemented, one third (29 per cent) of the long-term faculty in the School of Education did not report any publication activity. This is particularly striking because 11 members of this group had published journal articles prior to 1987.

For those faculty w h o do not publish, department chairs and deans will have to find alternative motivational strategies, including providing (a) systematic support for professional development activities that would enhance faculty member s ' capability to do research, (b) mentoring, and (c) assistance with obtaining external funds for research projects. Finally,

31


there is ongoing discussion about the desirability of modifying the faculty reward structure to provide equal recognition for significant accomplishments in an area other than research by allowing faculty to specify which of the four performance areas was to be emphasized each year. This would reduce the importance of research productivity, but only for those faculty providing convincing evidence of significant achievement in the teaching and/or service categories.

32

Appendix A University of Pittsburgh

School of Education

T o : School of Education Faculty and Research Associates

From: Kenneth F . Metz , Interim Dean

Date: 11 February 1991

Subject: Procedures for faculty merit and compensation review, Spring 1991

All indications from the University's central administration indicate that salary increases for faculty will again be based, at least in part, on merit considerations that emanate from our annual review and evaluation of each faculty member ' s performance. The period of review covers activity for the past year. Thus, each faculty m e m b e r in the School of Education is asked to prepare a dossier of their accomplishments during the past 12 months in preparation for a 1 July 1992 merit classification and compensation decision.

In keeping with the principle that faculty evaluations should have a c o m m o n base across initial hiring, annual review and promotion and tenure decisions, the categories for evaluation this year continue to be (1) research (intellectual achievement), (2) teaching, (3) professional activity, and (4) service (university and community). There are various subcategories within each of them. Current School of Education Promotion and Tenure Guidelines provide suggestions for different ways of documenting the quality as well as the quantity of accomplishments in each of the four main categories. A form is attached to facilitate reporting of activities during this past year.

33

Appendices

A . Reporting activities

In preparing a dossier for review, the faculty m e m b e r should bring together material which reflects accomplishments in each of the four appraisal areas described in the following. T o guide the review process at the Department and School level, the faculty m e m b e r should use the attached (or reasonably similar) form and include a personal statement summarizing achievements for the year under review as well as goals and plans. The statement should show h o w the kinds of specific activities and scholarly products completed, in process, or projected, relate to the candidate's goals and career plans.

1. Research and scholarship

Evidence of a productive and creative mind is sought in the faculty member ' s record which, in this category, normally includes copies of: (1) articles published reporting research completed, (2) books published that report research results, (3) monographs reporting research conclusions and supporting evidence, and (4) research reports submitted to granting agencies. Scholarly works properly included in this category are: (1) philosophical treatises tending to provide the framework for hypothesis development, (2) analytic studies offering n e w directions for inquiry, (3) theoretical propositions tending to guide investigation, (4) publications which show the potential of unique operational concepts, and (5) publications which invite experimentation on promising institutional processes. In jointly authored materials the faculty m e m b e r should identify clearly his/her contribution.

Consideration is not given to research and scholarly works which have been given credit on previous reviews unless major revisions have been made . The character or nature of the revisions should be made explicit by the faculty m e m b e r . In process work as well as unpublished manuscripts will normally be considered if they are presented in draft form.

34

Appendices

2. Teaching

A listing of courses taught by term with final enrollments as well as all independent studies conducted should be included in the dossier. Similarly, a listing of the number of advisees by level, their progress to degrees and certificates and/or their completion of degrees and placement following completion (when known) should be indicated.

Effective teaching is an essential criterion for the receipt of a salary increase. In this regard, the Faculty Council's approved Policy on Student Evaluation of Teaching specifies that: (1) every faculty m e m b e r has the responsibility to supply evidence of effective teaching in yearly evaluation and promotion and tenure dossiers; (2) it is strongly recommended that each faculty m e m b e r (part-time and full-time, tenure stream and nontenure stream) select at least one course per term in every academic term of teaching to be evaluated by students using the forms and procedures of the Office for the Evaluation of Teaching (OET)\ (3) the results in the report issued by the Office for Evaluation of Teaching will be sent only to the individual faculty m e m b e r requesting the survey; and (4) each faculty m e m b e r will select those reports resulting from student evaluations of teaching to be included in the dossier assembled for promotion and tenure, salary increases, or other personnel decisions. The selection of the course(s) to be evaluated is at the discretion of the instructor with the understanding that the same course should not be chosen repeatedly, but that, over time, a variety of courses should be evaluated. For Spring 1991 reviews, the faculty m e m b e r should include selected evaluations from Spring Term (91-2), S u m m e r (91-4), Au tumn Term (92-1) and Spring Term (92-2), if available. Also, keep in mind that original O E T reports should be provided rather than xerox reproductions or normative summary statements.

At present there is no formal process for obtaining documentation of teaching performance from a faculty member ' s colleagues. However, such pertinent and judiciously conceived documentation as m a y be submitted by the faculty member ' s colleagues will be considered. This documentation might deal with (1) the person's knowledge of his/her teaching

35

Appendices

field(s), (2) the degree to which knowledge is reflected in the design and execution of the person's courses, (3) the appropriateness of the person's course content and materials to the field in which he/she is teaching, and (4) observations of the person's classroom teaching.

Within the faculty member ' s statement of achievement and intent, comments on teaching might suggest a perspective for viewing the person's teaching efforts. For example, the statement might (1) explicate what the faculty m e m b e r perceives to be the goals of his/her teaching, (2) note exceptional or extraordinary circumstances which the faculty m e m b e r wishes reviewers to consider, and (3) indicate w h y the faculty m e m b e r selected a particular source of information to document teaching effectiveness. Included in this statement should also be information on the faculty member ' s performance as sponsor of degree candidates and/or individuals with w h o m the faculty m e m b e r has worked on research projects, tutorials, theses, dissertations, and so on.

In addition to data on teaching effectiveness, the faculty member ' s dossier should include some or all of the following: (1) course outlines, (2) positions occupied by the students w h o have completed graduate studies under the faculty member ' s sponsorship, (3) examinations given to each class listed as being taught by the faculty member , and (4) textbooks written by the faculty member .

3. Professional contributions and activity

In a professional school, demonstrated achievement in the competence appropriate to the faculty member ' s professional field and its activities should be recognized as a criterion for salary compensation. Data concerning professional contributions and activity m a y include: (1) joint activities with other members of the profession; (2) appointed or elected posts in professional organizations; (3) evaluations of colleagues in the profession; (4) accepted or invited papers delivered to learned groups; (5) editorial assignments for scholarly journals; (6) writing assignments requested by scholarly publications; (7) technical assignments requested by local, state, national and international educational authorities; (8)

36

Appendices

documented use of publications by other scholars; (9) technical assignments requested by foundations, governments, and other organizations; (10) consultancies resulting from public service; and (11) creative publications resulting from university service.

4. University and public service

Faculty are expected to play an important role in the administration and governance of the University. Evidence should be included in the dossier which demonstrates that faculty have participated actively and effectively in faculty governance and the formulation of Departmental, School and University policies. Similarly, service to community, state, national and international entities should be cited.

Data concerning University service should include: (1) Departmental committee service, (2) School committee service, (3) University committee service, (4) Departmental administrative service, and (4) University Senate service. Data concerning public service m a y include service to: (1) local school districts; (2) county educational authorities; (3) state educational authorities; (4) national and international educational authorities; (5) local, state, national and international organizations supporting education activities; and (6) professional organizations protecting the academic rights and obligations of educators.

В« T h e review process

In conducting the faculty reviews, each Department chair should adhere to the following procedures that are in keeping with the Principles, Goals and Recommendations for Faculty Salary Policy as stated by the University Senate Budget Policies Committee in April 1986, and the Faculty Safeguards in the Merit Review Process developed by the School of Education's Faculty Development and Compensation Committee

1. Each faculty m e m b e r prepares his/her annual report in accord with the above guidelines and submits it to the Department Chair.

37

Appendices

2. The Chair reviews all materials submitted by each faculty m e m b e r in the Department in accordance with the Faculty Titles, Roles and Merit Review Guidelines and, on the basis of an assessment of each person's work, places them into one of the following five categories of performance:

A . Unsatisfactory - for those whose performance does not meet the standards of satisfactory performance;

B . Satisfactory — for those whose performance meets the standards for satisfactory performance of the role of a faculty m e m b e r as outlined in the Handbook for Faculty (pp. 42-60). The satisfactory category recognizes those faculty members whose dossier demonstrates attainment across the four areas of research, teaching and scholarship, professional activity, and university and public service;

C . Merit -- for those whose performance is balanced and exceeds the standards for satisfactory performance in at least one area;

D . High Merit - for those whose performance exceeds the standards for satisfactory performance in several areas with outstanding performance in one or more categories, which must include research;

E . Exceptional Merit - for those whose performance exceeds the standards for satisfactory performance in most areas accompanied by exceptional research performance.

3. The faculty member , following the submission of a complete dossier, has the option of scheduling a meeting with the chair to discuss the results of his/Tier review and evaluation. At this time, the faculty m e m b e r has the opportunity to provide any new information that m a y bear on the assessment. This meeting m a y also serve as an annual

38

Appendices

planning meeting at which the faculty member , in conjunction with the Chair, defines the goals, tasks and assignments that will be addressed next academic year in terms of research and scholarship, teaching, professional activities, and university and community service. Based on the annual review and the outcome of the review meeting (if scheduled), the Chair prepares a letter of recommendation to the Dean with a copy to the faculty m e m b e r that includes the assessment of the faculty member ' s work and an indication of the recommended performance category.

4 . All materials submitted with each faculty member 's dossier and the chair's letter of recommendation for each person are submitted to the Dean. The two Associate Deans and the Dean independently review the materials, place the assessment of the dossiers within the context of the total School to m a k e certain that judgments are consistent across departments, and meet with each Department Chair to discuss the accomplishments of each faculty member . The Chair of Faculty Council or Council's designee will be present during the final reviews. The Dean, in keeping with the timing of the announced compensation increase by the University administration, communicates to each faculty m e m b e r the final results of his/her review. A faculty m e m b e r has the right to appeal the results of the review and any salary adjustments resulting from such an appeal are applicable beginning with the first month of the 1992 fiscal year.

Thank you in advance for your cooperation in this important process.

39

Appendix В University of Pittsburgh

School of Education

Suggested form for annual faculty review, academic year 1990/1991

Faculty are encouraged to utilize this form in preparing for the annual faculty review. The form is provided for faculty convenience and its several items correspond to the general categories of appraisal used in the School of Education and University. Should certain faculty members find this form inappropriate, they are encouraged to use some other sensible format to offer evidence that will enable an informed appraisal of their work and contributions.

N a m e

Department(s) ( %) Name

( %) Name

Salary source ( %) Name

( %) Name

Rank Number of years in rank

40

Appendices

1. Evidence of intellectual achievement (i.e. research and scholarship). Please attach copies.

A. Publications in press/published:

B. Manuscripts submitted/in preparation:

2. Teaching and related activity with students. Please attach syllabi and evidence of teaching effectiveness.

A. Classroom instruction Course number and title Term Enrolment

B. Directed study and sponsored dissertation research Course number and title Term Enrolment

С Student advisement. Please submit a copy of the office of the registrar or O S P S list of your student advisees registered during the last three terms.

D. Student committee membership

1. Master's degree committees

M e m b e r Chair

2. Doctoral degree committees

M e m b e r Chair

3. Comprehensive examination committees

M e m b e r Chair

41

Appendices

O f those masters and doctoral committees you indicated above that you chaired, please list the names of students completing degrees since the last reporting period:

N a m e Degree/term

Position after certificate completed completion

3. Professional activity

A. Papers presented to conferences. Please attach copies as appropriate.

B. Other professional/conference participation. Please specify the nature of the participation.

C . Consulting

D . Grants, fellowships, Awards and Honors. A s appropriate, please list source, title, purpose and amount.

4 . University service

A. University service

(e.g., committee membership, leadership roles)

B. School of education service

С School of education department service D . School of education departmental program or specialization

service

E. Non-university service

42

Appendices

5. Other evidence of accomplishment, information or comments

6. Please attach a current curriculum vitae and a statement outlining your achievements, plans and goals

Signature Date

43

References

Baird, L . (1986). "What Characterizes a Productive Research Depart ment?", Research in Higher Education, 25 (3), 211-225.

Bowers, J.K. (1989). "Issues in Developing a Faculty Evaluation System". Journal of Personnel Evaluation in Education, 3, 31-38.

Centra, J.A. (1989). "Faculty Evaluation and Faculty Development in Higher Education", pp. 155-179 in Smart, J.C. (Ed.), Higher Education: Handbook of Theory and Research, Vol. V . N e w York: Agathon Press.

Cresswell, J . W . ; Brown, M . L . (1992). " H o w Chairpersons Enhance Faculty Research: A Grounded Theory Study". Review of Higher Education, 16 (1), 41-62.

Crosson, P . H . (1983). Public Service in Higher Education: Practices and Priorities. A S H E - E R I C Higher Education Research Report N o . 7. Washington, D C : The George Washington University, School of Education and H u m a n Development.

Fox, M . F . (1992). "Research, Teaching, and Publication Productivity: Mutuality Versus Competition in Academia". Sociology of Education, 65 (4), 293-305.

Weidman, J .C ; Potter, E . ; Hyra, D . ; Tounkara, A . "Student and Faculty Development and Evaluation", pp. 32-36 in Spaulding, S.; Mauch, J.; Nyirenda, S.; Potter, E . ; Sabloff, P.; Weidman, J. (1991). Research on Higher Education in Developing Countries: Suggested Agendas and Research Strategies. Final Report, UNESCO-University of Pittsburgh Forum of Experts on Research on Higher Education. Paris: U N E S C O (ED-91/WS-29).

Weimer, M . ; Lenze, L.F. (1991). "Instructional Interventions: A Review of Literature on Efforts to Improve Instruction", pp. 294-333 in Smart, J.C. (Ed.), Higher Education: Handbook of Theory and Research, Vol. VII. N e w York: Agathon Press.

44

П Е Р publications and documents

More than 650 titles on all aspects of educational planning have teen published by the International Institute for Educational Planning. A comprehensive catalogue, giving details of their availability, includes research reports, case studies, seminar documents, training materials, occasional papers and reference books in the following subject categories:

Economics of education, costs and financing.

Manpower and employment.

Demographic studies.

The location of schools (school map) and sub-national planning.

Administration and management.

Curriculum development and evaluation.

Educational technology.

Primary, secondary and higher education.

Vocational and technical education.

Non-formal, out-of-school, adult and rural education.

Copies of the catalogue m a y be obtained from the H E P Publications Unit on request.

The International Institute for Educational Planning

The International Institute for Educational Planning (IIEP) is an international centre for advanced training and research in die field of educational planning. It was established by U N E S C O in 1963 and is financed by U N E S C O and by voluntary contributions from M e m b e r States. In recent years the following M e m b e r States have provided voluntary contributions to the Institute: Belgium, Canada, Denmark, Finland, Iceland, India, Ireland, Norway, Sweden, Switzerland and Venezuela.

The Institute's aim is to contribute to the development of education throughout the world, by expanding both knowledge and the supply of competent professionals in the field of educational planning. In this endeavour the Institute co-operates with interested training and research organizations in M e m b e r States. The Governing Board of the 1ГЕР, which approves the Institute's programme and budget, consists of eight elected members and four members designated by the United Nations Organization and certain of its specialized agencies and institutes.

Chairman: Victor L. Urquidi, (Mexico) Research Professor Emeritus, El Colegio de Mexico,

Mexico.

Designated Members: Arturo Núñez del Prado, Director, Latin American and the Caribbean

Institute for Economic and Social Planning, Santiago. Cristian Ossa, Director, Development Policy and Analysis Division, Department

of Economic and Social Development, United Nations. Visvanathan Rajagopalan, Vice-President and Special Adviser to the

President, The World Bank. Allan F. Salt, Director, Training Department, International Labour Office.

Elected Members: Isao Amagi (Japan), Special Adviser to the Minister of Education, Science and

Culture, Ministry of Education, Science and Culture, Tokyo. Mohamed Dowidar (Egypt), Professor and President of the Department

of Economics, L a w Faculty, University of Alexandria, Alexandria. Kabiru Kinyanjui (Kenya), Senior Programme Officer, Social Sciences

Division, International Development Research Centre, Nairobi. Tamas Kozma (Hungary), Director-General, Hungarian Institute for Educational

Research, Budapest. Yolanda M . Rojas (Costa Rica), Academic Vice-Rector, University of

Costa Rica, San José. Michel Ver nier es (France), Professor of Economic Sciences, University of Paris I

Panthéon-Sorbonne, Paris Lennart Wohlgemuth (Sweden), Director, Scandinavian Institute of African

Studies, Uppsala

Inquiries about the Institute should be addressed to: The Office of the Director, International Institute for Educational Planning, 7-9 rue Eugène-Delacroix, 75116 Paris, France.

T h e book

With the objective of improving faculty performance, the study to which forms part of an IffiP research programme directed by Bikas C . Sanyal, describes the rationale behind introducing a system for the assessment of members of a faculty of a well established university. It traces the complex process of implementing this system, the difficulties encountered, the ways most of these were overcome. Finally, it elucidates the benefits of introducing such a system of evaluation by focusing on h o w this rationalises the incentive structure of the faculty, h o w it increases faculty output and provides higher satisfaction among the students.

The case study is expected to generate important insights for those institutions of higher education which intend to better the quality of their education by improving faculty performance through the introduction of an assessment system.

T h e authors

Chairman of the Department of Administrative and Policy Studies at the University of Pittsburg until 1993, Professor John С W e i d m a n holds the U N E S C O Chair in Kenya. Author of a large under of books and papers related to management of higher education. H e has served as a consultant for several national and international agencies including UNESCO.

Formerly the Associate Director for Student Aid Operations at the University of Pittsburg Daniel C . M a y has been working as Administrative Associate of the School of Education of the same university since 1983.

Implementing a faculty assessment system: a...

Documents

Transcript of Implementing a faculty assessment system: a...