Findings from a National Findings from a National Longitudinal Study Longitudinal Study of the RISE Programof the RISE Program

P. Wesley Schultz

California State University, San Marcos

Presented at the 2009 NIGMS Council Meeting, January 23, 2009, Washington DC.

Findings from a National Findings from a National Longitudinal Study Longitudinal Study of the RISE Programof the RISE Program

Research Team:

Mica Estrada-Hollenbeck Richard SerpeAnna Woodcock David MorollaRandie Chance Paul HernandezMaria Aguilar Perla RivasVictor Rocha Brian McDonald

Minority Training Programs

NIH values diversity Underrepresentation is an important

issue◦ STEM and biomedical sciences

◦ Women in science

◦ Racial and ethnic representation (Black, Native American, Hispanic/Latino)

Many NIH-funded programs promote research careers among members of underrepresented groups

◦ Primarily through NIGMS-MORE

Do These Programs Work?

1. National-level Data◦ 80% increase in minority PhDs since 1977

62% increase in minority population during same period

2. Site-specific Data◦ Each NIH program submits yearly progress

reports, and longer-term evaluation results◦ Typically track their student success, but

what about a control group?

Do These Programs Work?

3. Large-Scale Evaluation Assessment of NIH Minority Research and

Training Programs (Phase III). Science, 2006. Surveyed students from 47 programs (1970 -

99) Substantial limitations in finding former

students 5,371 identified participants, of which data

was obtained from 739 (only 83 trainees funded as undergraduates--19 U*STAR, 31 Bridges, 33 COR)

Problems with Existing Data

1. Programs can “cherry pick” students who are likely to succeed

- These students are likely to succeed without the program (growing new talent versus harvesting abilities)

2. No control group

3. Long-term evaluations outside of funding scope

4. No way to examine the “mechanisms” of success

5. Retrospective accounts can be biased

Overview: The Science Study

Prospective, propensity matched control Longitudinal study of R.I.S.E. and MARC

students Participants from 25 programs nationwide Twice yearly surveys from students Matched control group

◦ For each R.I.S.E. or MARC student, we find a similar student who does not go through the program

◦ Matching variables: ethnicity, gender, major, GPA, intention to become a scientist, enrollment level (LD, UD, Grad)

◦ Secondary matching: age, parental education, community college transfer, English as first language

Overview: The Science Study

Longitudinal Panel

Entering Panel N=1,228 (Fall 2005)◦ 457 R.I.S.E./MARC◦ 457 Matched Control◦ 157 Other funded (and 157 matched)

Longitudinal Panel

Survey Data Collection

Data collected through web interfacewww.TheScienceStudy.com

Psychological constructs self-efficacy, scientific identity, task-goal orientation, value orientation, ethnic identity, integrative identity, satisfaction, self- esteem, etc.

Accomplishments/milestonespublications, presentations, research participation, conference attendance, degree completion, etc.

Perceptions of the academic environment mentorship, belonging, financial support and MTP experience

W0Fall ‘05

100%

W1Spring ‘06

84.7%

W3Spring ’07

78.0%

W2Fall ‘0678.9%

W4Fall ‘0775.4%

W5Spring ’08

76.0%

Survey coverage:93.1%

Survey Measures

Response rates

Questions we can (ultimately) answer

1. Does participating in the RISE/MARC program cause an increase in the likelihood that a minority student will pursue a career in the biomedical sciences? (summative)

2. What are the types of activities that students involved in RISE/MARC programs are exposed to?

3. Are there some types of students who benefit more from the RISE/MARC program than others?

4. Are there elements of the RISE/MARC program that are linked with student success? (Research experience, faculty mentoring, financial support, motivated peers, resume of accomplishments)

Preliminary Findings1. Basic effects (summative outcome)

• Intention to become a scientist• Graduation rates• Applications and acceptance to doctoral

programs

2. Program elements (mediators)• Financial support, research experience,

faculty mentorship

3. Reasons for success (process)• Self-efficacy, identity, values

Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10)

Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10)

C=1.58

Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

Baccalaureate Graduation

•W0 Juniors and Seniors.•MTP = Minority Training Program

Graduate School:Applications

* W0 Juniors and Seniors who have graduated with B.A./B.S. MTP = Minority Training Program.

Graduate & Medical School:Enrollment

* W0 Juniors and Seniors who have graduated with B.A./B.S. MTP = Minority Training Program.

10%13%

1%

2. Program Elements (Mediators)

Support to attend professional conferences

Faculty Mentorship

ResearchExperience

Paid Tuition & StipendsTutoring

Invited Speakers & Workshops

2. Program Elements (Mediators)

Note: Table reported at the 2007 NIH progress report meeting. Results based on survey responses from 25 RISE directors, and 457 RISE students.

2. Program Elements (Mediators)

• Mediation analyses of programs

• Focus on three program elements• Research Experience

• Faculty Mentorship

• Financial Support (hours spent in employment off campus)

Mediation: Research Experience

c=1.58**

c’= .68*

a=.36 ** b=2.38 **

Research Experience

RISE StatusIntention to Pursue

a Career in the Biomedical Sciences

Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .001Bootstrapped Indirect Effect: Mean = 0.68, CI99% = 1.31 to 0.45Sobel: Z = 4.67, p < .001

Note: Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. Analyses are based on students who were undergraduates (jr. or sr.) at W0. Propensity score (W0) used as time invariant covariate. RISE = students continuously funded, and MATCH = students never funded by any program and enrolled on a RISE campus. Research is any research experience ever during undergraduate education. Intention to pursue career as biomedical scientist.

Mediation: Faculty Mentor

c=1.58**

c’= 1.33**

a=.32** b=.78*

FacultyMentor

RISE StatusIntention to Pursue

a Career in the Biomedical Sciences

Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .001Sobel: Z = 2.08, p < .05

Mediation: Financial Support

c=1.60**

c’= 1.49**

a=-.26** b=-.42ns

FinancialSupport

RISE StatusIntention to Pursue

a Career in the Biomedical Sciences

Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01Sobel: Z = .85, nsFinancial support is hours spent working off campus

PROCESS MODEL FOR RISE OUTCOMESPROCESS MODEL FOR RISE OUTCOMES

Program Constructs Outcomes

Faculty Mentorship

(Student-Faculty Contact)

Conference (attending)

Cohort (peer support)

Research Experience

Financial Support

Motivation to Become a Scientist

-Learning motivation

- Performance motivation (approach)

-Performance motivation (avoid)

Identity as a Scientist

-Salience

-Commitment

-Integrative (with ethnicity)

Efficacy to Achieve

-Scientific Self Efficacy

-Self Esteem

Intention to pursue graduate education in the biomedical

sciences

Behavior

-GPA

- GRE/Other

-Schools applied to/accepted

-Degree(s) attained

-Professional career choice

Accomplishments

-Presentations

-Publications

Value of Scientific Community Objectives and Lifestyle

3. Process of Influence

3. Reasons for Success (process)

c=2.55 **

c’=2.28**

a=.28*b=.98**

Scientific Self-Efficacy

(rule)

Research Experience

Intention to Pursue a Career in the

Biomedical Sciences

Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01Bootstrapped indirect effect: mean = .27, CI99% .06 to .56Sobel: Z = 2.57, p<.01

3. Reasons for Success (process)

c=2.55 **

c’=1.80**

a=.56**b=1.34**

Scientific Identity

(role)

Research Experience

Intention to Pursue a Career in the

Biomedical Sciences

Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01Bootstrapped indirect effect: mean = .75, CI99% .33 to 1.23Sobel: Z = 4.37, p<.001

3. Reasons for Success (process)

c=2.51**

c’=2.24**

a=.24**b=1.08**

Scientific Values (value)

Research Experience

Intention to Pursue a Career in the

Biomedical Sciences

Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01Bootstrapped indirect effect: mean = .27, CI99% .01 to .60Sobel: Z = 2.43, p<.01

Key Points1. NIH programs for undergraduates (RISE and MARC) have a

strong effect on student intentions to pursue a career in the sciences.

2. RISE and MARC undergraduates have higher graduation rates, applications to doctoral programs, and enrollment in graduate programs (preliminary).

3. RISE works primarily because it provides students with research experience.

4.