Victor M. H. BordenAssociate Vice President, University Planning, Institutional Research &

Accountability (IU)Professor of Psychology (IUPUI)

Using Evaluation Research to Meet Expectations for Educating Students from

Disadvantaged Backgrounds

Scott EvenbeckDean, University CollegeProfessor of Psychology

Indiana University Purdue University Indianapolis (IUPUI)

Overview

► Introduction and Context► The Nina Mason Pulliam Legacy

Scholars Program♦ What is it and what does it entail?

► The evaluation model♦ Combining qualitative and quantitative

methods to optimize usefulness

► The impact of evaluation on program and student success

Introduction and Context► Four institutions invited by the Trust to write

proposals to support the high risk students as scholars♦ AZ: Arizona State University and Maricopa

County Community College District♦ IN: Indiana University Purdue University

Indianapolis and Ivy Tech Community College-Central Indiana

► Separate call for evaluation program♦ Qualitative Experts(Peg Bortner at ASU; Ken

Duckworth at IUPUI)♦ Quantitative Expert (Yours Truly)

The Scholarship Program► Providing educational opportunities to

individuals who would not normally receive traditional academic scholarships

♦ Adults, age 25 + with dependents in family unit♦ College-age youths and adults with physical

disabilities♦ Young adults, 18-25 yrs, raised in the child welfare

(foster care) system, responsible for own financial support

► Support provided♦ Money for tuition and fees, books and supplies♦ Annual living allowance (orig $2500, now $2750)♦ Coordinated services and support in a “cohort” model

The Evaluation Model

► Three components♦ One-on-one interviews with scholars, staff,

advisors♦ Annual Scholar survey: ratings of 11 service

components♦ Tracking of persistence and academic

progress of Scholars against matched comparison group

One-on-one interviews

► Initially interviewed every Scholar in first two years

► Moved on to Program Managers, Advisory Councils and some other involved constituents

► Intensive qualitative analysis (NUDIST/NVivo)

► Phased down as program matured and Scholar survey implemented

Scholar Survey► Started in fourth year► Ratings regarding importance to Scholar

success♦ 8 Nina Program-specific components:

• Program director; Relationship with other scholars; Scholar orientation; Academic advisement; Nina Scholar luncheon; Tutoring; Scholar meetings; Mentoring

♦ 3 Campus-wide services• Services for reentry students; Counseling services;

Services for students with physical disabilities► Open-ended comments for each item and

in general

Tracking System► Scholars tracked along with a “Matched”

comparison group♦ Comparison group derived from scholarship

eligible students who were not offered the scholarship but who subsequently enrolled

► IRB-approved contracts signed with each institution to share student records data annually♦ Fall request for enrollment information for new

scholars♦ End of year request for academic performance

information, including credits attempted, completed, and grades

Tracking System (cont)

► Stabilized over first three years with lessons learned♦ E.g., you can not create academic year by

combining fall and spring

Tracking Results

Tracking Results

Tracking Results – Multivariate

B Std. Error Wald Sig. Exp(B) Delta-P(Constant) 0.064 0.681 0.009 0.925 1.07Age -0.011 0.014 0.652 0.419 0.99 -0.2%Gender -0.083 0.331 0.063 0.802 0.92 -1.4%First year credits attempted 0.081 0.017 23.954 0.000 1.08 1.3%African American -0.359 0.208 2.973 0.085 0.70 -6.4%Other Underrep. Minority 0.197 0.219 0.807 0.369 1.22 3.0%

Scholar Status 0.808 0.274 8.684 0.003 2.24 10.0%Child of welfare system -1.020 0.370 7.620 0.006 0.36 -20.9%Physical Disability -0.233 0.348 0.447 0.504 0.79 -4.0%Inst 1 0.289 0.312 0.857 0.355 1.33 4.2%Inst 2 0.079 0.407 0.038 0.845 1.08 1.2%Inst 3 0.798 0.410 3.786 0.052 2.22 9.9%

Table A3. Logistic Regression of Control and Program Factors on Retention to the Second YearB Std. Error Wald Sig. Exp(B) Delta-P

(Constant) 0.064 0.681 0.009 0.925 1.07Age -0.011 0.014 0.652 0.419 0.99 -0.2%Gender -0.083 0.331 0.063 0.802 0.92 -1.4%First year credits attempted 0.081 0.017 23.954 0.000 1.08 1.3%African American -0.359 0.208 2.973 0.085 0.70 -6.4%Other Underrep. Minority 0.197 0.219 0.807 0.369 1.22 3.0%

Scholar Status 0.808 0.274 8.684 0.003 2.24 10.0%Child of welfare system -1.020 0.370 7.620 0.006 0.36 -20.9%Physical Disability -0.233 0.348 0.447 0.504 0.79 -4.0%Inst 1 0.289 0.312 0.857 0.355 1.33 4.2%Inst 2 0.079 0.407 0.038 0.845 1.08 1.2%Inst 3 0.798 0.410 3.786 0.052 2.22 9.9%

Table A3. Logistic Regression of Control and Program Factors on Retention to the Second Year

Intentional Program Development► Are we on the right track?

♦ Quantitative data as a dashboard♦ Qualitative feedback and narratives as a map

► Validation of holistic and intrusive approaches♦ Are we right?♦ Where are the boundaries

► Emphasis on cohesiveness, mutual support, building community

♦ Is the director a guru?► Educational goals revised (two-year to four-

year and four-year to grad school)  and Development of passport program

Evaluation as the “Third Leg”

► Emerging literature on best practices♦ “High Impact” programs

► Doing things right is as important as doing the right thing

► Evaluation drives organizational learning for doing things right♦ Including doing evaluation right