Post on 21-Jan-2016
Mixed Methods Dissertation Design Challenges and Opportunities: A Sequential, Explanatory Approach to Studying Students’
2/4 Transfer
Robin LaSota, PhD, University of WashingtonPost Doctoral Research Associate, University of Illinois Urbana-
Champaign (UIUC)Office of Community College Research and Leadership (OCCRL)
Higher Education Collaborative PresentationFebruary 25, 2014
College of Education at the University of Illinois Urbana-Champaign
Presentation Overview
• Evaluation Criteria – Mixed Methods Research• Dissertation Design Summary• Quantitative and Qualitative Strands
– Design Choices– Paradigm– Findings– Policy Inferences
• Analytical Reflections• Mixed Methods Analysis Revisited
Some Criteria for Evaluating Mixed Methods Research
• Clearly stated research purpose(s) and reasons for design choices about methods and mixing
• Alignment between research methods and purpose
• Conclusions offer the “value added” from mixed methods approach
• Conclusions include an analysis of study legitimacy, limitations, and paradigms used for mixed method analyses
Reasons for Mixing Methods
1. Triangulation2. Complementarity3. Development (of sampling, instrumentation,
measurement decisions, etc.)4. Initiation (of theoretical perspective, new questions,
contradictions)5. Expansion of breadth and range of inquiry
From: Greene, Caracelli, and Graham (1989) in Creswell and Clark (2011), p. 62.
My Research Purpose
• To offer “complementary findings regarding strategies, policies, and factors including improvements to students’ upward transfer, particularly for students most at risk of not transferring and attaining a baccalaureate degree” (p. 251)
• Each strand of inquiry (quantitative and qualitative) has separate research questions and purposes
A First Area of Inquiry
Q1.1 How do student behaviors, community college characteristics, and state policies and conditions influence students’ upward transfer probability?
Q 1.2 How do these factors, policies, and conditions influence upward transfer probability, particularly for low-income and first-generation community college students?
Student, College, and State Factors Influencing 2/4 Transfer
Academic and MjSocial Mj
Integration, MjCollege Major, MjCollege GPA... Mj
State Articulation and MjTransfer Policies and State Mj
Characteristics Mj
Public Two-Year Colleges' MjCharacteristics Mj
Six-Year Transfer MjProbability from MjTwo to Four-Year Mj
Institution Mj
Student MjBackground Mj
Characteristics Mj
Precollegiate MjTransfer Mj
Intentions Mj
Factors Affecting MjCollege Mj
Persistence (Work MjHrs, FT vs. PT) Mj
A Second Area of Inquiry
Q2.1 What are promising practices in colleges and states aimed at improving students’ upward transfer, and how may they constitute a system of support?
Q2.2 How do leaders engage in ongoing
innovation around these practices?
Explanatory Sequential Mixed Methods Design
Multi-
level modeling analysis of BPS
and supplemental data 2003-
09
Case studies of 6
colleges in 3
states
QUANT/QUAL
Integration:
1) Using QUANT
analysis to guide QUAL
sampling
2) Cross-reference
claims from QUANT and
QUAL3) Use QUAL
to complement and extend
QUAN.
Research Paradigm – Quantitative Strand
• To “quantitatively measure theoretically and empirically-derived factors (from prior research) found to produce a greater likelihood of students’ transfer from public two-year to four-year institutions in a recent, nationally representative sample, particularly for students who are low-income and first-generation to earn a bachelor’s degree.” (pp. 7-8)
Sampling Design Choices from BPS
• N=5010 community college students; weighted N=1,528,900
• Students not co-enrolled in two or more colleges• BPS nationally representative longitudinal population
survey of all postsecondary entrants• BPS not representative of states, colleges, or CC entrants• State and community college factors investigated build
upon existing literature
• State articulation and transfer policies? Very little, if any influence (Roksa, Kienzl, Goldhaber & Gross)
• State cooperative agreements? Maybe. (Kienzl)
• Community college practices? Depends. Perhaps not much. (MDRC)
• Community college expenditures? Slight/student services expenditures (Gross and Goldhaber). None (Stange).
• Community college smaller size, higher faculty-to-student ratio? Yes. (Bailey et al., Gross & Goldhaber)
• Degree of college mission stratification (emphasis on transfer-oriented programs vs. non-transfer oriented, e.g. health/vocational/technical). Influential. (Dougherty)
• Proximity and selectivity of nearest public four-year institution? Maybe. (Rouse)
Which State and Community College Factors May Influence Transfer?
Rationale for Multi-Level Modeling Methodology
Results of Unconditional Model or Intra-Class Correlation –• State location – explains 2% of variance in 2/4 transfer probability • Primary college attended – explains 6% of variance• Student characteristics – explains most of the variance
Therefore, used multi-level logistic regression –• Randomly varying intercepts and slopes between colleges and states for
– Low-income, first generation– First generation, not low income– Planned to transfer at time of entry– Declared major in health/vocational/technical field
Positive Predictors Associated with Upward Transfer Probability
Se-ries1
0.00 0.05 0.10 0.15 0.20 0.25
0.010.040.050.06
0.070.08
0.090.09
0.120.21
0 = 50/50 Probability of 2/4 TransferConditioned on Factors in the Model
Primarily Full-Time
Planned to Transfer at Entry
Aged 15-19 at Entry
Worked 1-19 Hrs/Wk on Average
GPA in First Year (tenths)
Sports Participation Often or Sometimes
STEM, Humanities, Education Major
Gross State Product (standardized)
Academic Advising Often or Sometimes
CC Transfer Out Rate
Negative Predictors Associated with Upward Transfer Probability
Series1
-0.25 -0.20 -0.15 -0.10 -0.05 0.00
-0.01
-0.02
-0.04
-0.04
-0.14
-0.15
-0.19
0 = 50/50 Predicted Probability of 2/4 TransferConditioned on Factors in the Random Effects Model
Primarily Part Time
First Generation, Low Income
First Generation, Not Low Income
CC Pct Health Voc Completions
Took Any Remedial Education
Unemployment in CC's County
Health, Vocational, or Prof/Tech Major
College Characteristics: Association with Upward Transfer
• Proportion of associates’ degree completions in health/vocational fields (neg., p<.10)
• College transfer-out rate (2% increased odds of transfer) in regression without analysis of random effects by slope
• County-level unemployment (neg., p<.10)
Not sig. = i.e. per-student expenditures for instruction or student services, distance to nearest public four-year institution, distance to nearest non or less-selective four-year institution, faculty-to-student ratio, community college enrollment size, percent of full-time faculty, percent of full-time students
State Policies’ Association with Students’ Upward Transfer
Policy Components- Transfer data reporting- State transfer incentives- State transfer guide- Transferable general education
curriculum
- Statewide cooperative agreements- Common course numbering- Statewide articulation/transfer
policy
• Main Effects Model, Random intercepts only, no varying slopes+ 35% higher 2/4 transfer odds: State with one standard deviation higher Gross State Product Per Capita in 2003
• None of the State Articulation and Transfer Policy Components explained variance in 2/4 transfer probability.
Regression Results: Slopes for Sub-Populations that Vary by College and/or State
Low-Income, First Generation:• Higher gross state product• Common course numbering• College transfer-out rate
First Generation, Not Low Income:• Higher Gross State Product• Common Course Numbering
Planned to Transfer (vs. Not Transfer Intending):
• College transfer-out rate
Health/Vocational Major (vs. business/undeclared):
• State articulation/transfer policies not sig.
• Transfer-out rate not sig.
Random and Fixed Effects Modelshowed that these factors moderate 2/4 transfer probability
for these populations.
Green = Sig. at p<.05; Gray=Not Sig, p>.05
Findings:Quantitative Inquiry Strand
• Affirmed prior research about ambiguous or unknown effects of state transfer and articulation policies
• Offered new evidence about the role of state common course numbering in increasing first-generation students’ transfer
• Influential college-level factor – College mission focus; college’s transfer-out rate
• Full-time attendance and transfer intention are particularly influential student factors
Some Implications: Quantitative Strand
• Promising areas for policy intervention, esp. in high schools – Help students create specific plans for obtaining a
bachelor’s degree aligned in a specific field and outline a transfer pathway
– Promote continuous full-time attendance and advising with incentives and accountability
– Widely promote available state resources and policies for improved transfer
Rationale for Case Study Design
• Goal: To explore and identify possible state policy actions and college policies or practices that enhance student 2/4 transfer probability
• Structure analysis for meaningful contrasts relative to the goal– States and Colleges with Higher Transfer vs. Average Transfer Rates
(within their state)– Policy Innovative States in Articulation and Transfer– Colleges Engaged in Data-Use and Innovation– States with significant CC sector and states & colleges with student
populations of interest
Research Paradigm – Qualitative Strand
• To offer “a more nuanced and detailed picture” of college and state practices supporting students’ upward transfer, that “built on and complemented my quantitative findings” (p. 101)
• Use framing literature to “interpret institutional cultures, data use, and innovation in practices at the six colleges”…and “to help pinpoint the context for innovation…[as well as] transfer-related practices.”
State Case Selection: Florida, Georgia, and Washington
• Used OLS regression to find states performing above average in transfer, controlling for state and student population characteristics
• Considered prior research on policy innovative states in transfer and articulation
• Chose states with a considerable proportion of postsecondary students enrolled in two-year colleges and with racial/income diversity
College Selection: Above-Average and Average Performer
• Used OLS regression to find colleges performing above average in transfer, controlling for college and student population characteristics
• Consulted State Higher Education Executive Officers (SHEEO) from each state and Aspen Prize Top 120 data
• Used SHEEO advice and college’s participation in Achieving the Dream indicators of data-use and innovation
Qualitative Methods
•Interviews with state policy officials in articulation and transfer (N=20)
•Interviews with college administrators, faculty, and student affairs staff (N=110)
•Individual interviews and focus groups with students (N=49)
•N=179 overall
Semi-structured, one-hour interviews
Analytic Strategy: Analytic Memo Writing and Data Synthesis
Findings: Qualitative Inquiry Strand
Academic and MjTransfer Advising Mj
Practices Mj
Data Use for MjDecision Support Mj
Leadership Support Mjand Structure for Mj
Innovation Mj
State Policy, Influential MjNetworks and Labor Market Mj
Trends Mj
Low-Income, First- MjGeneration, Under- Mj
Represented MjStudents Mj
Findings – Advising in Above-Average Performers
Transfer not a universal outcome or push for all students…
College-level systems of support for transfer generally constrained…
Above-average colleges generally have:
•Academic leaders who champion students’ transfer and successfully engage others in this work
•Mandatory student advising models
•Student affairs staff dedicated to coaching students’ on transfer
Findings – Advising in Above-Average Performers
Above-average colleges also tend to have:
•Faculty contracts which include student advising hrs.
•Faculty and staff engaged in planning out-of-class supports and enrichment experiences for students that aid transfer
•Campus supports for TRIO and similar STEM programs for low-income, minority, and first generation students
•Key Support for Stronger Advising: Active communication/coordination with public and private four-year institutions within major fields by administrators and faculty
Findings – State Policy as a Context for Colleges’ Innovation
Common Course Numbering: Lessons Learned from Florida
Moderating positive influence of common course numbering (CCN) for first-generation college students from quantitative inquiry…• CCN Proxy for a more robust transfer policy context?
• CCN built from communication across lower and upper division faculty and programs
• Florida: CCN in place for 30 yrs; created when 2 yrs and 4 yrs were governed in one system
Some Implications:Qualitative Inquiry
States: • Incentives and support for college-level innovation• Support for measuring innovation effectiveness• Build transfer into performance accountability
Colleges: • Collaborative problem-solving re: transfer• Broad implementation of personalized learning & transfer advising• Incentives to be transfer champions
States and colleges: • More efficient, accessible processes to using data for decision support about
students’ transfer
Analytical Reflections - Quantitative
• Data Limitations– BPS measures of academic and social integration– State policy measures binary coding– No adequate measure of policy strength for the period– Available college-level data mostly not predictive of transfer
• Not a causal inference multi-level model• Does not examine reasons for stopping out or mixed
attendance – e.g. role of financial aid
Analytical Reflections - Qualitative
• Examined broad scope of practices affecting transfer probability (from pre-college to graduation check/final term advising) rather than one or two specific innovations
• Used analytical memo writing not software-based coding methodology
• Inductive approach to claim formulation rather than deductive hypothesis-testing
• Different framing literatures inform each strand, complementary analyses, not necessarily integrated
Mixed-Methods Analysis Revisited
• “Complementary strengths, overlapping weaknesses” of the two strands
• Strands answer different questions, however, from the QUAN findings, QUAL cases consider:– Role of common course numbering– Students’ experience with academic advising and
colleges’ work to reform it– Additional policy supports for improved upward transfer
(not represented in original coding)
With Appreciation• To all the participants in my study• To Debra Bragg and OCCRL for post-doc support• To NISTS for the honor of the award and presentation with you• To my chair, Bill Zumeta• To my co-advisor, Marge Plecki• To my committee members:
– Mike Knapp– Bob Abbott– Jennie Romich
• To my fellow doctoral students• And to IES and AIR for funding
Sponsored by the US Department of Education, Institute of Education Sciences (#R305B090012)and the Association of Institutional Research Dissertation Grant
Questions/Discussion
Follow up contact information:rlasota@illinois.edu
Office of Community College Research and LeadershipCollege of Education at Illinois
occrl@illinois.eduoccrl.illinois.edu