Benchmarking for Continuous Improvement Rafiq Dossani RAND Corporation 10 October 2015.

Benchmarking for Continuous Improvement

Rafiq DossaniRAND Corporation10 October 2015

ABOUT RAND

UNIVERSITYCONSULTANCY

RA N DTHINK TANK

National Security Research DivisionArroyo Center Project AIR FORCE

Pardee RANDGraduate School

Justice, Infrastructure, and Environment

EducationHealthLabor and Population

RAND EuropePentagon image: DoD photo by Master Sgt. Ken Hammond, U.S. Air Force.

THE PARDEE RAND GRADUATE SCHOOL

New Orleans

Boston

Pittsburgh

Washington, D.C.

Cambridge

Brussels

Santa Monicaheadquarters

campus

Canberra

RAND’s annual revenue:$263,000,000

U.S. federal, na-tional security

U.S. federal, health

U.S. federal, other

U.S. state, local, and other gov-ernment agen-cies; interna-tional clients

UniversitiesPrivate sector

FoundationsDonors

Objectives of Benchmarking

• Provide students with a guide to college (institution) and discipline selection

• Provide colleges with metrics with which they can function in a ‘competitive environment’ and ‘improve their ranking’. (Minister, HRD, NIRF Document, 2015)

• Provide policymakers with assessments and recommendations on the education system’s capacity to meet employment targets and other national goals.

Outcomes Relevant to Students • Which college adds the most long-

term value to someone with the entering student’s academic ability?– Long-term Value:

• Career monetization – Measure: Present value of lifetime

earnings – estimated by sampling graduates’ earnings at early, mid-career and late-career stages

• Multiple-job preparedness. The median number of number of jobs of current graduates in the US is estimated at 7 jobs during the earnings lifetime– Measure: Interviewing graduates at times

of job change to determine whether the change improved lifetime earnings prospects. » Proxy: sampling late-career

graduates’ job status and earnings progression

• Multiple-role preparedness within a field (ranging from greater specialization to greater generalization at various stages)

– Measure: Similar to multiple-job preparedness.

• Less: Entering student’s long-term value if she entered a ‘similar college’.

• Most such measures are costly

• The typical proxy measures are the student’s academic score at entry, academic score at exit, and earnings at graduation

Outcomes Relevant to Colleges• Student’s goal of value-addition• Rising quality of students over time

– Measure: Academic profile of entering students over time• Rising research capacity of institution over time

– Measure: Publications in high-impact journals, citations, real-world impact• These three goals may be inconsistent with each other

– E.g., A college catering to lower-tier students may find it costly to maintain a remediation programme as the percentage of such students declines or as it invests more in research. For instance, the University of California system prices remediation at high prices with the expectation that students will use lower-quality, cheaper community colleges to complete such courses. This enables UC to preserve resources for higher-end students and research

• Proxy measures include: Faculty qualifications, faculty experience, faculty consulting

Outcomes Relevant to Policymakers

• System diversity:– Adequate supply of different levels of the quality of education

provided, to match profile of students’ academic abilities • Measure: Inter-institutional quality diversity mapped against entering

student ability

– Adequate supply of specializations to meet demands of the marketplace over time• Measure: Intra- and nter-institutional specialization diversity, and

mapping to market needs

• Rising quality with time, with respect to national needs and global benchmarks; reduction of lower-tier share with time– Measure: Longitudinal quality measures

General Comments on Measures• Measures should not be highly correlated. E.g., if percentage of

students obtaining jobs through campus placement is highly correlated with the average salary at graduation, using both measures to measure lifetime earnings is about equivalent to doubling the weight on earnings

• Direct measures are better than proxies. E.g., if faculty qualifications and publications are highly correlated, the more direct measure of research output, i.e., publications, is preferred. – Measures of inputs and throughputs are less preferred than measures of

outputs and outcomes• Long-term outcomes should be measured by more stable rather than

less stable measures. E.g., in a young institution, citations – which take several years to trail in, may be a less preferred measure than publications.

Summary of Measures & RelevanceStudents Colleges Policymakers

Earnings

Entering student’s academic score

Exiting student’s academic scoreQuality of research

System diversity

Longitudinal inter-institutional quality measures

What does NIRF offer?• Ranking Parameters (Category A

institutions):– Teaching, Learning and Resources

30%• Faculty-Student Ratio 9%• Faculty Qualifications and

Experience 9%• Libraries and Labs 9%• Sports and Extra-curricular

Activities 3%

– Research, Professional Practice and Collaborative Performance 30%• Publications/Faculty 9%• Citations/Publication 9%

• IP/Faculty 4.5%• Collaborative Publications & IP 3%• Footprint of Projects and

Professional Practice 4.5%

– Graduation Outcome 15%• Examinations 4.5%• Placement, Higher Studies and

Entrepreneurship 7.5%• Earnings 3%

– Outreach and Inclusivity 15%• Outreach 3.75%• Out of state recruitment 3.75%• Women 3%• Disadvantaged students 4.5%

– Perception• Peer rating 10%

What does NIRF offer? – Output and Outcome Measures

• Ranking Parameters (Category A institutions):– Teaching, Learning and Resources



Activities 3%








What does NIRF offer? – Input and Throughput Measures




Activities 3%








What does NIRF offer? Input/Throughput & Output/Outcome




Activities 3%








Relevance of NIRF Measures

Outcomes to be measured ->

Value-addition

Long-term Student Quality

Long-term Research Quality

System Diversity

System Quality

F:S Ratio - 9

Fac Qualif & Experience - 9

Library/Lab - 9

Sports/EC - 3

Pubs - 9

Citations - 9

IP – 4.5

Collab Res – 3

Consulting–4.5

Exam Perf – 4.5

Placement -7.5

Earnings - 3

Outreach/Inclusivity - 15

Peer rating - 10

Comments on NIRF

• Value-addition is not measured adequately since entry value is not measured

• “Market signals” include placement, earnings, publications, citations and consulting. An

additional market signal that is relatively easily measurable and reliable is demand by entering students (the fill-rate).

• Student needs are relatively unfulfilled.

Measurement Issues

Challenges -> Data Source Validation, if any Reliability

F:S Ratio - 9 College Random audit Medium

Fac Qualif & Experience - 9 College Random audit Medium

Library/Lab Expenditure - 9 College Random audit High

Sports/EC Spending & Performance - 3

College Random audit Low

Pubs/Faculty - 9 Global agency High

Citations/Pub - 9 Global agency High

IP/Faculty – 4.5 College Random audit Medium

Collaborative Research – 3 Global agency High

Consult/Research Fund –4.5 College Random audit Medium

Public/Univ Exam Perf – 4.5 Boards, College Random audit Low

Placement -7.5 College Low

Earnings - 3 College, Employers LowOutreach/Inclusivity/PD - 15 College Low

Peer rating - 10 Survey of all stakeholders (excl. students)

Low

Reliability of NIRF Measures


Value-addition



System Diversity

System Quality

F:S Ratio - 9


Library/Lab - 9

Sports/EC - 3

Pubs - 9

Citations - 9

IP – 4.5

Collab Res – 3

Consulting–4.5

Exam Perf – 4.5

Placement -7.5

Earnings - 3


Peer rating - 10

EDU-RAND Ranks


Value-addition



System Diversity

System Quality

Exit scores – 50%

Publications/Citations (Composite Index) – 27%

Faculty Qualif/Experience – 11%

Fill-rate – 12%

Measurement

Challenges -> Data Source Validation, if any Reliability

Exit scores – 50% Domestic Agency

High


Global Agency High


Website College Survey Medium

Fill-rate – 12% Website College Survey Medium

Reliability of EDU-RAND Measures


Value-addition



System Diversity

System Quality

Exit scores – 50%



Fill-rate – 12%

Comparison of Ranking Systems

NIRF Times Higher Education ARWU EDU-RAND

F:S Ratio - 9 F:S – 4.5

Fac Qualif/Experience-9 Disting Faculty - 22 Fac Qualif/Exper - 11

Library/Lab - 9

Sports/EC - 3

Pubs/Faculty - 9 Pubs/Faculty – 4.5 Qlty journals - 44 Publications – 13.5

Citations/Pub - 9 Citations/Pub – 32.5 Cited Faculty - 22 Citations -13.5

IP/Faculty – 4.5

Collaborative Resch – 3

Consultancy/Research Fund – 4.5

Industry Income – 2.5Research Funding – 5.25

Pub/Uni Exam Perf – 4.5 Ph.D.s/F – 6:Ph.D./ UG -2.25 Disting Alumni - 12 Exit Scores - 50

Placement -7.5

Earnings - 3

Outreach/inclusion - 15 Internl faculty students -

Peer rating - 10 Teaching survey – 15Research survey – 19.5

Fill rate - 12

Conclusions

• Students, colleges (institutions) and policymakers need different benchmarks.

• Benchmarks can be hard to measure, forcing the use of proxies

• NIRF proposes the use of a number of unreliable proxy measures, highly correlated measures, input/throughput rather than output/outcome measures, and does not adjust for institutional stage of growth or overall system stability.

• NIRF is more detailed than other measures, but without adding relevance or reliability.

• Simpler, more objective measures, such as EDU-RAND, may offer a viable alternative.

Additional Slides

Summary of NIRF Measures & Relevance

Outcomes to measure ->

Earning Score(All)

Entry Score (Student + College)

Exit Score(Student + College)

Research Score (College + Policy)

System Diversity (Policy)

Longit Quality (Policy)

F:S Ratio - 9


Library/Lab - 9

Sports/EC - 3

Pubs - 9

Citations - 9

IP – 4.5

Collab Res – 3

Consulting–4.5

Exam Perf – 4.5

Placement -7.5

Earnings - 3


Peer rating - 10

Benchmarking for Continuous Improvement Rafiq Dossani RAND Corporation 10 October 2015.

Documents

Transcript of Benchmarking for Continuous Improvement Rafiq Dossani RAND Corporation 10 October 2015.