Michele J Hansen, Ph.D.

Executive Director of Research, Planning, & Evaluation

mjhansen@iupui.edu

Lauren Chism, M.S.,

Director of Themed Learning Communities

lchism@iupui.edu

Daniel J. Trujillo, M.S.

Qualitative Research Analyst

djtrujil@iupui.edu

University College

The Synergy of and Readiness for High Impact

Practices During the First-Year of College

Presentation Overview

2

• Literature Review• Research Setting & Interventions• Research Questions • Research Methods • Results• Discussion • Limitations and Guidelines for Future Investigations • Questions!

Presentation found at: http://research.uc.iupui.edu/

3

Literature Review

• Summer Bridge Programs

• First Year Seminars

• Learning Communities

Summer Bridge Programs

4

• Summer programs designed to assist students’ transition to college• Generally include attention to study skills, writing and math.• Positive outcomes have been associated with program participation• More research needed

First Year Seminars

5

• Common intervention for beginning students          

•  Program Objectives (Padget & Keup, 2012)• Developing academic skills• Developing a connection with the institution• Providing an orientation to campus resources & services

• First Year Seminars are often linked with increases in retention (Pascarella & Terenzini, 2005)

• Mixed results in the literature regarding the impact on GPA• Higher GPA (Friedman & Marsh, 2008) (House, Rode & Xiao, 2008) (Schwartz &

Grieve, 2008)• Higher GPA only in select groups (Dalgreen, 2008) (Hansen, Williams & Chism, 2008)• No impact on GPA (Rogers, Eglsaer, Muehsam, Cailouet & Kan, 2008)

 

Learning Communities

6

• Cohort of students enrolling in linked courses, typically connected through an interdisciplinary theme

 

Learning Communities

7

• Most institutions have learning communities programs, but not all students are able to participate:

• 62% reported having learning communities (Barefoot, 2002)• 18% of NSSE participants reported participating in learning

communities. (NSSE, 2011)

• Increases & retention & GPA

• Learning communities are designed to engage students in learning, retention gains are not the purpose, but a welcome byproduct. (Lardner & Malnarich, 2008)

 

Conducting Research with Students Participating in Multiple Interventions

• Studies often lack a control group (Kezar, 2000)

• Random assignment is rare.• No gains in persistence for 16 randomly assigned learning non-traditional

learning community students at a proprietary technical institution.(Goldberg & Finkelstein, 2002)

• Gains in retention for first year seminars (Strumpf & Hunt,1993)

• Students participate concurrently in multiple interventions• While most learning community programs include a first-year seminar,

participation in first-year seminars is not accounted for in many studies examining the impact of learning communities on grade point averages (Andrade, 2007-8).

• Huber (2010) has completed some preliminary research looking at the effects of multiple high impact practices on graduation rates based on National Survey of Student Engagement (NSSE) data.

Research Setting• Ranked 3rd "up and coming" American

universities and Recognized for Learning Communities & the First Year Experience (U.S. News and World Report)

• 8th best public college in the Midwest (Forbes) • Student population of just over 30,000

students• Over 250 degree programs from both Indiana

& Purdue Universities, guided by the Principles of Undergraduate Learning

• University College serves as entering unit. • Majority of FT, FT students commute to

campus (73%) • Many FT, FT are first generation college

students (41%)• Many FT, FT receive a Federal Pell Grant (44%)

9

Intervention Descriptions

Common elements of 3 interventions(Summer Bridge, First Year Seminars and Themed Learning Communities)

• Designed for entering freshmen• Cohorts of 25 students• Instructional team

• Faculty member• Academic Advisor• Student Mentor• Librarian

10

Summer Bridge Program

Essential elements:

• 2 week program prior to fall classes introduced to collegiate-level expectations for writing, mathematics, communications, critical thinking

• become more acquainted with the campus, and learn important study skills.

More information available at: http://bridge.uc.iupui.edu/

Summer Bridge: Providing a Sense of Readiness

• 421 First-Year students participated in 2010 (240 subsequently participated in a TLC).

• Completed an anonymous end-of program questionnaire.

• Included both Quantitative and Qualitative Items.

• Allowed for students’ opinions and perceptions of the program to be gathered.

Campus Navigation & Locating Resources

Note: Responses provided on a Likert scale ranging from1 = Strongly Disagree to 5 = Strongly Agree

Response Means2010

*Find my way around campus 4.63

*Campus resources available to me (e.g., Writing Center, Math Assistance Center, Learning Center

4.50

*Feel connected to IUPUI 4.47*Beneficial interacations with librarian 4.14

*Find what I need at the library 4.11

Participating in the Summer Bridge improved my ability to or understandanding of….

1 5

Diversity Awareness & Appreciation

Notes: Responses provided on a Likert scale ranging from 1 = Strongly Disagree to 5 = Strongly Agree.

Response Means

2010

*Appreciate cultural & social diversity 4.49

*Instructional team members that were committed to promoting an environment that respects and celebrates diversity

4.42

*Activities and or classroom discussions that incorporated diversity awareness and appreciation

4.35

*Developed and appreciation of social and cultural diversity

4.29

*Had a serious conversation with students of a different race or ethnicity

4.10

Participating in the Summer Bridge improved my ability to or understandanding of…. 1.00 5.00

Academic Success: Critical Thinking, Writing, Math, & Public Speaking

Notes: Responses provided on a Likert scale ranging from 1 = Strongly Disagree to 5 = Strongly Agree

Response Means2010

*Faculty expectations of students 4.50

*Succeed academically 4.43

*Feel able to meet the demands and expectations of college

4.37

*See multiple sides of an issue 4.17

*Critically examine ideas and issues 4.14*Communicate my thoughts in speaking 4.16*Think critically about what I read 4.09

*Do research projects effectively 3.98

*Do well in math courses 3.77

Participating in the Summer Bridge improved my ability to or 1.00 5.00

Examples of Actual Student Comments

.

• “I think learning about all the resources available to students”.

• “Learning college etiquette, exploring diversity, study skills”.

• “Lives up to its name, bridge between high school and college”.

• “Helped get me into the college mindset and show me college level skills”.

• “Expectations of how college was going to be liked were changed”.

• “Learning what to expect in college and how to be an organized student”.

First Year Seminars

Essential elements:

• 1-3 credit course

• Introduces academic skills, key campus resources, and creates opportunities to connect with faculty and staff.

Themed Learning Communities (TLCs at IUPUI)

Essential elements:

• 3 or more linked courses

• Interdisciplinary theme & connections

• Learning beyond the classroom

More information available at: http://tlc.iupui.edu

First Year Seminar

A.

Summer Bridge

B.C.

TLCD.

Students Participating in Multiple Interventions and High Impact Practices

2010 Students Participating: A=1018, B= 142, C= 400, D=240

1. Is participation in high impact practices and a summer bridge intervention significantly positively related to higher levels of academic success (first-year GPA and one-year persistence rates), even when considering student characteristics?

2. Does the synergy of multiple high impact practices (themed learning community with an embedded fist-year seminar) contribute to students’ academic success levels more than high impact practices in isolation or no participation, even when considering student characteristics?

Research Questions

3. Does participation in a summer bridge program prior to participation in multiple high impact practices interventions (a TLC with an embedded first-year seminar) contribute to academic success levels more than participation in high impact interventions without Summer Bridge, even when considering student characteristics?

4. Is participation in a Summer Bridge-TLC significantly and positively related to higher first year grade point averages after accounting for the confounding effects of self-selection?

Research Questions (continued)

Quasi-experimental design Hierarchical Linear Multiple Regression Hierarchical Logistic Multiple Regression Instrumental Variable Analysis in an attempt to

account for self-selection Participants: 2028 FT, FT regularly admitted

students in 2010 cohort

Research Methods and Data Analyses

Many students would have been denied access to programs (ethics).

Logistical issues with registration and making sure that only students in the “experimental group” could enroll in the sections (administrative).

Political pressures coming from legislators that low-income, first-generation students should participate in academic support interventions to help them make successful transitions to college (political).

Random Assignment Not Feasible

Outcome Variables: • First-Year GPA • One-Year Retention Rate

Interventions (entered in Step 2; dummy coded) • First Year Seminar (no TLC or Summer Bridge)• Themed Learning Community-FYS (no Summer Bridge)• Summer Bridge (no TLC-FYS)• Summer Bridge – TLC-FYS

Covariates (entered in Step 1):• Academic Preparation (HS GPA, SAT scores)• Gender (Female 1, Male 0) • Low Income (Pell Grant 1, No Pell Grant = 0) • Proxy for Student Motivation (weeks since admit date)

Variables (Multiple Regressions)

Causal inference challenging in quasi-experimental designs - possibility of hidden bias.

Hidden bias may exist as a result of failure to control for unobservable factors.

Threats to internal validity such as self-selection. PSM not used because large samples are required,

hidden bias may remain because matching only controls for observed variables (to the extent that they are perfectly measured) (Shadish, Cook, & Campbell, 2002).

Why IV Analysis?

A variable z is called an instrument or instrumental variable for the regressor x in the regression model y = x+u if (1) z is uncorrelated with the error u; and (2) z is correlated with the regressor x.

Can solve the problem of omitted variable bias – IV can help solve the problem of missing or unknown variables when random assignment not possible.

Instrumental Variable Analysis

• The instrument must be correlated with the endogenous explanatory variables (Summer Bridge –TLC Intervention).• The instrument cannot be correlated with the error term in the explanatory equation. The instrument cannot suffer from the same problem as the original predicting variable (test of overidentifying restrictions, such as Sargan test).

Fundamental IV Assumptions

1. Stable Unit Treatment Value Assumption (SUTVA) – “spillover effects”

2. Random assignment3. Exclusion restriction4. Nonzero average causal effect of

instrument on treatment5. Monotonicity

(see Angrinst & Pischke, 2009 and Porter, 2012)

IV Five Assumptions

• Weak instruments may produce inconsistent and inefficient results.

• Valid and reliable instruments very difficult to locate.

Bane of IV Analyses

Instrument Endogenous Variable or Treatment

Dependent Variable

Study Authors

Miles from College

Attend Community College

BA Degree Completion

Long & Kurlaender, 2009

Draft lottery Military Service Earnings Angrist, 1990

Randomized Assignment

Program Participation

College Outcomes

Angrist, Lang, & Oreopoulos, 2009

Season of Birth Years of education

Earnings Angrist & Kreuger, 1991

Class size rules class size Test scores Angrist & Lavy, 1999

Examples of Instruments Used

• Campus Housing• Distance from Campus (based on zipcode)• HS Class Size • Applied for Financial Aid• Twenty-First Scholar Participant

Instruments Considered

Confidence in Timely Degree Completion

Placement in English Composition I

Current Study Instruments

Met two fundamental assumptions. Strongly correlated with SB-TLC Robust F statistic was 15.28 (df = 7, 2020; p < 0.001).

This F value was above the threshold of 10.00 recommended by Stock and Watson (2007).

The results of Sargan and Basmann tests of overidentifying restrictions were not statistically significant, indicating that the model was appropriately specified and the instruments were not correlated with the error term.

Evaluating the Instruments

“All instruments arrive on the scene with a dark cloud of invalidity hanging overhead. This cloud never goes entirely away, but researchers should chase away as much of the cloud as they can.”

Murray, 2010

Word On Instruments

Hierarchical Multiple Regression: Cumulative First Year GPA (N=2028)

Variable b SE b β

Step 1 HS GPA 1.04 .06 .41***

SAT Score .00 .00 .12***

Student Motivation .01 .00 .07***

Low Income -.24 .04 -.12***

Gender .11 .04 .06**

Step 2 HS GPA 1.04 .06 .41***

SAT Score .00 .00 .12***

Student Motivation .01 .00 .06**

Low Income -.24 .04 -.12***

Gender .10 .04 .05**

Summer Bridge-TLC .29 .08 .10***

FYS-TLC .25 .07 .10***

Summer Bridge (no FYS-TLC) .22 .09 .06**

FYS (no Summer Bridge or TLC) .09 .06 .05

R2 = .271 for Step 1: ∆R2 =.013 for Step 2 (p <.001). ***p<.001, **p<.01, * p<.05

Results Hierarchical Multiple Regression: One-Year Retention (N=2028)

Variable B SE Odds Ratio

95% CI Wald Statistic

p

Z H.S GPA .50 .06 1.64 [1.45, 1.86] 61.50 .000

Z SAT Score .14 .06 1.15 [1.01, 1.30] 4.81 .028

Z Student Motivation .17 .06 1.18 [1.06, 1.32] 8.95 .003

Low Income -.14 .11 .87 [ .70, 1.08] 1.64 .201

Gender -.22 .11 .81 [ .64, 1.01] 3.55 .060

Z H.S GPA .51 .06 1.66 [1.46, 1.88] 62.08 .000

Z SAT Score .14 .06 1.15 [1.01, 1.30] 4.71 .030

Z Student Motivation .13 .06 1.14 [1.02, 1.27] 5.05 .025

Low Income -.16 .11 .86 [ .69, 1.07] 1.97 .160

Gender -.22 .12 .80 [ .64, 1.00] 3.70 .054

Summer Bridge-TLC 1.34 .23 3.83 [2.43, 6.05] 33.40 .000

FYS-TLC .74 .19 2.09 [1.45, 3.01] 15.50 .000

Summer Bridge (no FYS-TLC)

.91 .26 2.47 [1.47, 4.16] 11.71 .001

FYS (no Summer Bridge or TLC)

.53 .16 1.70 [1.24, 2.32] 11.11 .001

Two Stage Least Squares Results for the IV Model

Dependent Variable: Summer Bridge-TLC Participation

Coefficient

Std. Err.

Constant 0.08 0.089

HS GPA -0.02 0.017

SAT Score 0.00 0.001

Student Motivation 0.01 0.001

Low-Income -0.02 0.015

Gender 0.06*** 0.015

Composition Self-Placement 0.13*** 0.014

Planned Complete Degree Timely Manner .03* 0.012

*p < .05, **p < .01, ***p<.001

Two Stage Least Squares Results for the IV Model

Dependent Variable: First Year GPA

Coefficient

Std. Err.

Constant -1.94*** 0.201

Participated in Summer Bridge-TLC 1.05** 0.309

HS GPA 1.08*** 0.060

SAT Score 0.01*** 0.001

Student Motivation 0.01* 0.002

Low-Income -0.26*** 0.041

Gender 0.05* 0.045

High Impact Practices

“When I am asked, what one thing we can do

to enhance student engagement and increase student success? I now have an answer: make it possible for every student to participate in at least two high impact activities during his or her undergraduate program, one in the first year, and one taken later in relation to the major field. The obvious choices for incoming students are first-year seminars, learning communities, and

service learning” (George Kuh, 2008)

• No positive effect for FY GPA once background characteristics were accounted for and considered in conjunction with the other interventions in the hierarchal linear multiple regression model.

• Positive effect on one-year persistence.• Vital component of TLC and SB• Instructional team model• Designed help students adjust, make connections,

career development and exploration.

First-Year Seminar

• TLC intervention made a more substantial contribution to GPAs than the Summer Bridge intervention or the standalone FYS.

• Synergy of multiple high impact practices (TLC with an embedded FYS) contribute to students’ academic success levels more than high impact practices in isolation or no participation, even when considering student characteristics.

Themed Learning Community and Synergy of High Impact Practices

• Summer Bridge intervention significantly positively predicted retention rates and cumulative first-year GPAs.

• May be an ideal early intervention strategy for counterbalancing any deficits that students have resulting in a lack of readiness to begin college.

• Broad-based intervention not restricted to particular groups such as low-come students from underrepresented minority groups.

Summer Bridge

• Summer Bridge-TLC contributed to academic success levels more than other interventions.

• Possible that the Summer Bridge intervention helped prepare students and provide a sense of readiness for the engaging and powerful pedagogies.

• Summer Bridge intervention may have provided the foundation of support and community that allowed students to fully reap the benefits offered in their TLC-FYS experiences.

Providing a Readiness for High Impact Practices: Summer Bridge-TLC

• Generalizability • Only Two Short Term Outcomes (FY GPA and One-

Year Persistence) • Results indicate if effective, but not why - what

components of TLCs (service learning, interdisciplinary theme, integrated assignments, sense of belonging and community) produced positive outcomes.

• Limitations of IV analysis prevent ability to make casual inferences.

Limitations

Treatment difficult to standardize. Faculty own curriculum. TLC program really consists of multiple

interventions as faculty teams collaborate on identifying unique interdisciplinary theme for their section.

Meeting students diverse needs and notion of “one-size does not fit all.”

Limitations in Post-Secondary Educational Contexts

• Multi-institutional studies.• Investigate more outcomes such as learning, graduation,

and degree attainment. • Mixed-Methods (qualitative investigations of program

implementation strategies to supplement quantitative methods).

• Variation within program studies. • Consider random assignment for pilots (new untested

program that would be small enough not to deny tx to large groups of students).

• Consider RA to different types of tx rather than not providing services/programs or waitlist controls.

Future Investigations

Questions!