Monroe Community College Data Driven Retention Strategies for Online Students Dr. Jeffrey P....

Monroe Community College

Data Driven Retention Strategies for Online Students

Dr. Jeffrey P. BartkovichMarie J. Fetzner

February 21, 2004

Monroe Community College 2

I. Introduction and Welcome

• Welcome and presenter introductions

• Brief audience survey• Handouts• Session evaluation


Presentation Agenda

• 12:30pm – 1:30pmIntroduction, Overview and Presentation of Study #1

• 1:30-1:40pm Break

• 1:40pm – 2:20pmPresentation of Study #2

• 2:20 – 2:30pm Break

• 2:30 – 3:15pm Research to Inform Practice

• 3:15 – 3:30pmQuestions and Discussion


Purpose of Presentation

• Provide an overview of two online student retention/withdrawal studies The problem, the research, the findings and the

applications

• Demonstrate retention strategies and policy• Discuss a research agenda to inform

practice Theory development, defining retention,

managing services and student-to-student advice


II. MCC Overview

• MCC is part of SUNY Founded in 1961; 13,326 FTE in 2003

• Joined SUNY Learning Network (SLN) in 1997

• Spring 2004 MCC SLN Summary Data Headcount = 3,434; Online FTE = 328.43 Courses = 86; Sections = 121

• Sharing in the development and delivery of SLN courses


MCC Online Courses and Sections

Fall 1997 to Spring 2004Source: MCC Academic Affairs and Educational Technology

86

778187

7371

55

41

2513

4

83 80 85

121

112

109

113

9097

69

49

32

13

4

110 104 110

0

10

20

30

40

50

60

70

80

90

100

110

120

130

140

FA97

SP98

FA98

SP99

FA99

SP00

FA00

SP01

FA01

SP02

FA02

SP03

FA03

SP04

Semester

0

10

20

30

40

50

60

70

80

90

100

110

120

130

140

Courses

Sections


MCC Online Program Support

• MCC is western NY SLN faculty training site

• MCC online faculty supported via The Monroe Model SLN and local campus participation Cross divisional membership Comprehensive on-site support

• Students supported via SUNY HelpDesk and MCC online and on-site resources


MCC Educational Technology Services (ETS)

Organization

CIO

Communications &Network Services

Computing Services Library ServicesInstructional Technologies

Telecommunication &PC Support

Distance Learning

Electronic Learning Centers


III. MCC Online Student Retention Study #1

Characteristics and Attitudes of Non-Retained Online Students

Office of Educational Technology ServicesMonroe Community College

Dr. Jeff Bartkovich and Marie Fetzner


The Research

• The Problem/Research Question• Method

Study limitations

• Descriptive Statistics• Correlations

Significant findings

• Conclusions and Applications to Practice


The Problem

Why are certain MCC online students (those with grades of F/W) not succeeding in their online courses?


Online Retention Study Method

• Archival Data Analysis Grades and demographics for online

courses retrieved from student records database

• Student Survey Survey created and administered to

sample of students who received an “F” or “W” in an online course for the Fall semesters in 2000, 2001 and 2002


The Survey

• Basic demographic data• Questions derived from

Online Faculty MCC Student Retention Survey E-resources


Survey Instrument

• Total of 45 questions in three areas Expectations at time of registration Satisfaction at time of withdrawal Reasons for withdrawal

Likelihood of enrolling in another online course


Research Limitations

• Specific Sample of convenience Problems with generalizability (n=201) Difficulty in contacting students who do not

successfully complete the course

• General Orientation procedures and other online

student services evolve over time Course design and faculty experience

improve constantly


Basic Student Demographics

OL and Site-based, By Percent

Spring 2002 Fall 2002

OL Site-based

OL Site-based

Female 68 55 71 55

Male 32 45 29 45

Minority 20 25 19 24

Non-Minority 80 75 81 76

Less than 20 18 27 15 30

20-24 33 33 35 32

25-29 16 11 15 11

30 and over 33 29 34 27


Archival Data Analysis

MCC Online vs. On-site• Grade Distribution• Grade Rates• Success Rates in Matched (ftf vs.

same section online) Courses


Overall Grade DistributionOL vs. Site-based, By

Percent

Spring 2002 Fall 2002

Grade OL Site-based OL Site-based

A 34.5 32.8 34.0 28.7

B 20.5 26.1 19.7 26.2

C 10.0 15.3 10.6 15.2

D 3.9 5.0 3.7 5.1

F 12.9 8.2 14.1 7.4

W 16.3 11.4 14.5 10.7


Grade RatesOL vs. Site-based, Fall

Semester

1999 2000 2001 2002

OL N 970 1,417 1,916 2473

C and Higher

69.7% 65.8% 65.3% 63.6%

F/W 24.8% 26.4% 29.7% 30.2%

MCC N 43,779 45,235 48,288 52,428

C and Higher

70.3% 69.7% 70.7% 70.7%

F/W 20.0% 21.3% 20.6% 20.5%


Success Rates in Matched Online and Site-based

Courses

On-campus, n = 16,291

Online, n = 1,719

% C or Better PercentagePoint

DifferenceOnline Site-based

Total 64.2 69.6 -5.4

Full-timePart-time

56.472.3

70.366.7

-13.95.6

1st time FT1st time & at risk1st time & not at risk

39.747.266.3

72.266.674.0

-32.5-19.4-7.7

Under 25 years of age25 years of age or older

54.575.2

67.875.7

-13.30.5


Success Rates in Matched Online and Site-based

Courses

On-campus, n=16,291 Online, n = 1,719

% C or Better Percentage

Point Difference

Online

Site-based

Part-time 72.3 66.7 5.6

1st time, PT & not at risk 73.5 66.4 7.1

Taking Eng, History, Math, Speech, Comm, Music, HVAC

51.6 65.3 -13.6

Not Taking Eng, History, Math, Speech, Comm, Music, HVAC

74.1 73.7 0.4


Retention Sample

MCC vs. Retention Sample • Gender• Age• Ethnicity


MCC, MCC SLN and Sample Profile

Gender and Age

2002-03 MCC Profile

MCCSLN

Profile

Sample Profile

FemaleMale

54.7%45.3%

70.8%29.2%

62.2%34.8%

Under 2020-2425-2930-3435-4445-5960 and over

31.8%32.2%11.0% 7.6%11.2% 6.0% 0.2%

14.3%36.9%14.0%11.6%16.7%6.5%0.0%

14.3%42.9%15.7% 8.6%11.4% 7.1% 0.0%


MCC, MCC SLN and Sample Profile Ethnicity

2002-03 MCC Profile MCC SLN Profile

Sample Profile

BlackHispanicAmer. Ind.NRAAsianUnknownWhite

16.2% 5.1% 0.6% 0.5% 3.2%0.0%74.4%

11.0%4.4%0.4%0.3%2.1%0.0%81.7%

12.9% 5.5% 0.0% 0.0% 3.5% 4.5%73.6%


First-time vs. Experienced Online Student Sample by Gender

• First-time online students (n = 132; 68%)• Experienced online students (n = 62;

32%)

0

10

20

30

40

50

60

70

80

90

FemaleMale

ExperiencedFirst-time

82

19

5043

n = 194 7 Unknown


First-time vs. Experienced Online Student Sample by Ethnicity

0

10

20

30

40

50

60

70

80

90

100

Black Hispanic Asian White

% First-time% Experienced


Expectations* at Time of Registration

% First-Time Students

% Experienced Students

Can Begin Anytime 41.4 18.8

Online Easier than On-Campus

40.4 22.3

Less Homework than On-Campus

40.0 20.0

Faculty Interaction Required 35.6 15.0

Need Basic Computer Skills 28.3 14.1

Participate Independently 18.9 35.0

*Definitely what I expected + Sort of what I expected


Satisfaction* at Time of Withdrawal



Own Performance in Course 30.0 14.0

Technical Help with the Course

28.6 13.5

Registration/Orientation 26.4 9.8

SLN in General 23.4 13.9

Directions Provided by Faculty

22.8 14.0

Directions to Get Started 10.9 15.1

*Somewhat satisfied + Extremely satisfied


Explanation for Non-Success

in Online Course

Top Ten Factors% First-

Time Students


Lack of Motivation 46.3 24.4

Course Taking too Much time

43.0 18.8

Instructor’s Teaching Style

43.0 18.4

Too Many Technical Difficulties

41.9 19.8

Got Behind/Couldn’t Catch Up

41.7 22.1


Explanation for Non-Success in Online Course

Top Ten Factors (Con’t)



Too Much Reading 41.4 20.9

Signed Up for too Many Courses/Had to Drop

41.1 20.4

Course too Unstructured 41.0 21.1

Not Interested in Subject 40.8 20.3

Couldn’t Handle Study Plus Other Responsibilities

38.9 21.6


Number One Reason for Non-Success—Ranked

Order

1. Got behind and couldn’t catch up2. Too much reading3. Course taking too much time4. Course too unstructured5. Too many technical difficulties


Number One Reason for Non-Success—Ranked

order

6. Not interested in subject matter7. Course too difficult8. Signed up for too many courses9. Couldn’t handle school with other

duties 10. Didn’t know where to go for help


Likelihood to Take Another Online Course

SurveyNot a

chance/not likely

It’s possibleSomewhat likely/very

likely

Fall 2000 31.8% 11.6% 56.5%

Fall 2001 58.1% 12.9% 29.0%

Fall 2002 55.9% 11.8% 32.3%

3 Year Total 48.5% 12.2% 39.3%


Likelihood to Take Another Online Course by Age

Age RangeNot a

chance/not likely

It’s possible

Somewhat likely/very

likely

<20 43.59% 15.9% 40.6%

20-24 34.4% 15.6% 50.0%

25-29 46.4% 14.3% 39.3%

30-34 64.1% 7.7% 28.2%

35-44 63.6% 0% 36.4%

45-59 50.0% 0% 50.0%

60 and over N/A N/A N/A


Likelihood to Take Another Online Course by Gender

Not a chance/not

likelyIt’s

possible


likely

Female 52.4% 8.1% 39.5%

Male 40.9% 19.7% 39.4%


Likelihood to Take Another Online Course by Ethnicity

EthnicityNot a

chance/not likely

It’s possible


likely

Black 48.0% 16.0% 48.0%

Hispanic 45.5% 18.2% 36.3%

Asian 49.2% 1.6% 49.2

White 41.3% 11.7% 40.0%


General FindingsArchival Analysis

When compared to site-based courses at MCC• Online students are more likely to

earn a grade of C or better if they are >25 years of age and are part-time students

• First-time, full-time online students are least likely earn a grade of C or better


General FindingsSurvey Analysis

Levels of Satisfaction• First-time students more satisfied at

time of withdrawal in general, and with their own performance

• Experienced students less satisfied with Registration procedures SLN in general Directions from faculty



Expectations• First time students more likely to expect

Fluid beginning and end dates Less homework Online easier than site-based courses

• Experienced students more likely to expect Limited faculty interaction Independent participation in course



Expectations• Online format expectations not accurate• Level of online interaction with faculty and

other students not accurate• Technical expectations (needed PC and

typing skills) not accurate

Percentage of W/F students’ likelihood to take another online course is decreasing• Approximately 1/3 are likely or somewhat

likely



Main reasons for withdrawal• Got behind and couldn’t catch up• Course had too much reading, took too

much time, was too unstructured and too difficult

• Had too many technical difficulties and didn’t know how to get help

• Not interested in subject matter and dropped course to cut course load

• Couldn’t handle school, family, work, etc.


Likelihood to take another online course•SLN in general (.604)•Interaction with other students (.518)•Directions provided by faculty (.491)•Directions to get started (.483) •Interaction with the faculty (.428)•Technical help with course (.408)

Significant Positive Correlations

(GPA and credit hours not significant with likelihood to take another course)

Significant at 0.01 level)


Significant Positive Correlations

• Lacked PC skills and Lacked typing skills (.970) Too many tech difficulties (.741)

• Lack of access to PC and Lacked PC skills (.766) Financial problems (.731) Too many tech difficulties (.688)

• Course too difficult/lacked PC skills (.629)

(Significant at 0.01 level)


Significant Negative Correlations

Likelihood to take another online course• Didn’t like the online format (-.569)• The course was too unstructured

(-.485)• Felt too alone, not part of the class

(-.435)• Didn’t know where to get help (-.324)(Significant at 0.01 level)


Factor Analysis Summary

• Factor analysis conducted Data converged into four key

groupings in seven iterations Further discussion on this

analysis will be conducted later in this presentation


Advice from the Literature

• The course design/level of interaction• The experience of the faculty• The level of technical support• A student pre-course orientation• Manage student expectations upfront• Professional development and training

for faculty• Standardized course management

system


Summary of Key Findings

• Retention of online (W/F) MCC students is impacted by these characteristics: First-time (vs. Experienced) online students Amount of previous higher education

credits for full-time online students Student age (<25) Lack of PC skills and lack of access to a PC


Research Informing Practice

Applications • Online student orientation CD project

Development and distribution of the CD Phone calls made to first-time online students

• On-site student orientations implemented* Development of Ten Myths Videostream

• Expansion of PC access for MCC studentsWireless laptops for check-out in LibrariesLearning Center support for online students

*An analysis was conducted on online student performance outcomes for those students who attended on-site MCC orientations


Research Informing Practice

• Additional materials development SLN Student Awareness Sheet

• Retention strategies included in online faculty course development trainings Emphasis placed on pre-course activities

and first week of course interactions

• Promote self-paced free technology training to MCC students

• Preliminary discussion of MCC policies Forwarded to Academic Leadership Council


Next Steps

• Conduct further analysis to investigate the relationship between online student reasons for non-success, and satisfaction variables for successful online students

• Investigate students’ technology readiness

• Continue evaluation of services and activities that were implemented as a result of the retention study


Summary and Questions

Questions on retention survey #1

Time for a break!


IV. MCC Online Student Retention Study #2

An Analysis of SLN Students’ Performance and Differentiation

Office of Institutional ResearchMonroe Community College

Angel Andreu


The Study Research

• The Problem/Research Question• Observations• The Data• Analysis• Conclusions and Policy Implications


The Problem

It is observed that the failure and withdrawal (F/W) grades for online students are increasing in number and percentage.


The Observation

Percentage of F/W Grades, Fall Semester

1998 1999 2000 2001 2002

Online* 19 25 26 30 30

College 17 20 21 21 21

*N in 1998 was 409; 2002 was 2,473


The Research

Previous research identified characteristics of students at risk of F/W grades in online courses. These were:

• First-time online students• Full-time students with less than 30

earned credits• Students under the age of 25• Minority students


The Research

The MCC “Andreu” study affirmed these characteristics and looked specifically at:• Time of registration• Basic academic skills


The Data: Time of Registration

• Only 7% of the “C or better” students registered during the first week of classes, compared to 16% of the “F/W” students

• Students who registered during the first week of classes had a 50/50 chance of earning a grade of C or better

• The odds of getting C or better decreased by 13% for each week closer to the first week of classes


The Data: Age

• 65% of the F/W grades were among students under the age of 25

• Between the ages of 21 and 22, a student had a 50/50 chance of earning a C or better


The Data: Age

Probability of Earning

An SLN GPA of C or Better

Figure 7

AGE, Years

605040302010

Pre

dic

ted

pro

ba

bili

ty

1.00

.90

.80

.70

.60

.50

.40


The Data:Age and Time of

Registration

• Age and time of registration are each significant contributors to grade performance

• Their interaction is not significant• The best chance of getting a C or

better Register 5 weeks before the start of the

semester Be age 25 or above


The Data: Experience(i.e. number of accumulated credit

hours)

• The rate of F/W grades between first and second year students is significant

• Experience is not significant when combined with age or time of registration


The Data: Part/Full Time Status

• Part/Full time status is not significant

• The interaction of status and age was significant


The Data:Part/Full Time Status

Estimated Marginal Means of SLN GPA

Figure 10

Age Group

30 & Over25 - 2920 - 24Less Than 20

Est

ima

ted

GP

A M

ea

ns

3.2

3.0

2.8

2.6

2.4

2.2

2.0

1.8

1.6

1.4

Status

Full-Time

Part-Time


The Data:Basic Academic Skills

• MCC’s ACCUPLACER tool The variables used included reading

and sentence scores The metrics are placement scores at

the college English level (78 for reading, 83 for sentences)


The Data:Basic Academic Skills

• For students to have a 50/50 chance of earning a C or better in an online course, their basic academic skills should be at the college English level


What Does the Analysis Show?

Factors important to success in online courses:• Time of registration• The age of the student• The academic preparedness of the

student



Factors in the decision model• College is enrollment driven• The online program is supporting

enrollment growth• The demographics of enrollment

growth for the College are the same demographics of non-success in online courses



Policy Factors• Should policy account for motivation• At what “odds of failure” do we restrict

enrollment• Should online growth be allowed to

continue to support college enrollment growth

• What is the balance between student success and online enrollment growth


Policy Development 1

• In all cases, a student with an ACCUPLACER score below 78 (reading) and/or 83 (sentence) is prohibited from taking an online course at MCC.

• This prohibition is lifted at such time that the student successfully completes the Transitional Studies coursework appropriate to address the deficiency.

• All online sections will be closed to registration one week before the start of class. Qualified students (see definition on next slide) who wish to enroll after this date will be allowed to “green slip” in until the College’s ‘Add’ deadline.



• Qualified students include mature students with a record of success in college-level coursework. Student must also have satisfactory ACCUPLACER scores to have successfully completed the Transitional Studies coursework appropriate to address any deficiency. Students who have successfully completed an online course previously are also considered qualified. Note that data indicate that students 22 years old or older have a significantly greater chance of success in online coursework.



Considerations• Forget policy, make it a course

prerequisite• Must be programmable• Cannot rely upon faculty involvement

for exceptions due to availability• Does resolution of the online problem

become a standard



Draft X• Students in remedial English courses

are prohibited from enrolling in online courses

• All registrations for online courses are closed on the first day of classes



Current Policy • Registration for online courses closed

week before the first day of classes• Late entry by permission


Summary and Questions

Questions on retention survey #2

Time for a break!


V. Research to Inform Practice:

An Online Research Agenda

MCC’s Online Research Agenda: Level 1 – Descriptive• Basic demographics• Basic production• Basic enrollment• Basic finance



MCC’s Online Research Agenda: Level 2 – Evaluative• Basic performance• Student and faculty satisfaction• Integration with campus IR• Modify existing surveys “to fit”



MCC’s Online Research Agenda: Level 3 – Comparative • Statewide benchmarks• National benchmarks• Complete surveys-–be a case study

MCC’s Online Research Agenda: Level 4 – Theoretical• External – in the literature• Internal – on the campus


Research to Inform Practice:SUNY Research

Reasons for Taking an OL Course*

Reasons for Taking Course Online• Conflict with Personal Schedule 49%• Family Responsibilities 23%• Distance or Transportation 10%• Other 10%• Course Not Offered on Campus 5%• Interest in Technology/Internet 3%

*Retained Students


SUNY ResearchReasons for Taking an OL

Course MCC vs. all other SUNY

Students* Primary Reason MCC* SUNY*

• Personal Schedule Conflict 49% 31%• Family Responsibilities 23% 16%• Distance or Transportation 10% 20%• Other 10% 12% • Not Offered On Campus 5% 15%• Interactive Technology 3% 7%

*Retained Students


Research to Inform Policy:Theory Development

Four major barriers to persistence in distance learning (Garland, 1993)• Situational – students’ general

environment (social, economic, family)• Institutional – college’s programs, policies,

and procedures• Dispositional – student’s personal

background• Epistemological – academic environment

(course, pedagogy, prerequisites)


Theory Development 2

1. Classified the 22 reasons for non-persistence according to Garland’s four barriers

2. Conducted Principle Components Factor Analysis

3. Analysis and Definition of the Factors


Research to Inform Practice:

On-campus Orientation

Question

Is there value to an on-campus orientation course for online performance?



On-Campus Orientation Results • Most students felt they had the necessary skills

to be successful• Skill preparation is not related to performance • Students who attended the on-campus program

were more likely to earn a C or better C/Better F/W

On-campus Orientation 78% 18%Online Orientation only 64% 30%



To Continue the Course• Practice Demonstration Course• Structure not Skills• Market the grade differential• Keep faculty involved in the program

to provide “authentic experience”



Student Advice

Question to online students who received F/W grade on their online course:

“What advice would you give a fellow student considering registering for an online course?”


Student Advice

Results – Top Five Topics• Be ready for independent learning.

Must be self-motivated.• Once you get started, stay on track.• Online courses require lots of reading,

homework and research. Get ready for the demands.

• Must develop a schedule to go online, must manage your time.

• Talk to other people and to other students before you take the class, and during the class


Student Advice

Practical Use of Online Student Advice• Integrate into brochures, orientation

seminars• Do mock testimonies for video

displays • Share with Counseling and Advising

staff• Share with Faculty


Research to Inform Practice:Managing Services

• Manage the Expectations Additional pre-start information sharing Additional orientations offered

• Manage the Support Services Greater integration with support team Additional points of access to services

• Manage the Academics Faculty course design issues and training Emphasis on multiple interactions


Managing Services

Manage the Expectations• Welcome letter from Academic Vice

President has been expanded• Face-to-face student orientations offered*

• MCC Online Orientation CD distributed

• Phone calls made to all Fall 2002 online students and 1st-time Spring 2003 online students

*additional analysis conducted


Managing Services

Manage the Support Services• Student Services web page

expansion• Revisions made to the service

gateway to online learning web page• Brochures at Records and

Registration and at the Counseling Center


Research to Inform Practice:Managing Services

Manage the Academics• Retention strategies integrated into

faculty trainings• Discipline-specific retention studies• Discussions on other academic support

—online tutoring, writing centers, etc.• Pilot test CourseSpace (use of online

template for web course enhancement)



Defining Retention

Question: How do you define retention?

It depends:• Research–two basic presentations

As an enrollment statistic As a performance indicator

• Policy Be consistent, clarify your statistics


Defining Retention and Withdrawal Rates

All SLN Courses Fall 2002Annual 2002-03

First day enrollment 1851 4154

First week enrollment 1857 4225

Census day enrollment

1714 3847

Last day enrollment 1716 3832

Retention as an enrollment statistic


Online Retention Rates

Fall 2002 1st Week Census Day Last Day

1st Day 1.02 .94 .95

1st Week .92 .92

Census 1.00

All 2002-03 1st Week Census Day Last Day

1st Day 1.02 .93 .92

1st Week .91 .91

Census 1.00


Defining Retention and Withdrawal Rates

Retention as a performance measure(All SLN Courses)

Grade Fall 2002Annual

2002-2003

A to D 1,253 2,755

F, W, Other 463 1,077


Withdrawal Rates

Fall 2002F&W

Grades

Spring 2003F&W

Grades

2002-2003F&W

Grades

All MCC 20.5% 19.6% 19.5%

Online Only 30.0% 29.5% 28.1%


Next Steps:Things to Consider

• Services • Policies• Definition of “Retention” for data

collection• Replicate studies at other

institutions


Summary

• Conclusions• Questions• Discussion• Completion of Session Evaluation

Forms


Contact Information

Dr. Jeffrey P. [email protected]

Marie J. [email protected]

MCC Web Page: www.monroecc.edu

MCC Online Learning page:http://www.monroecc.edu/depts/distlearn/index.htm

Monroe Community College Data Driven Retention Strategies for Online Students Dr. Jeffrey P....

Documents

Transcript of Monroe Community College Data Driven Retention Strategies for Online Students Dr. Jeffrey P....