Assessing Student Success Initiatives · 2017. 11. 15. · 2015 • Achievement Analytics •...

Assessing Student Success Initiatives:

Challenges and Results

September 19, 2017

Julio E. MorenoAchieving the Dream

Agenda: About Columbus State

Starting point and Integration with IR

Student Initiatives Challenge

Student Success Evaluations

Developing the Methodology: A Holistic Approach

Our Student Success Initiative Results

Tying It All Together: “The Big Picture”

Questions and Discussion

About Columbus State: Established in 1963 as Columbus Technical Institute

Main campus in downtown Columbus, Ohio– Delaware Campus and Regional Learning Centers

Enrollment of more than 26,000 students in central Ohio counties

About Columbus State:

63% white, 19% Black or African-American (Fall 2015)

71% attend part-time, 29% full-time

53% female

61% are age 24 and under (and growing)

35% Pell-eligible

Joined Achieving the Dream in 2012

Named an AtD Leader College in September, 2015

Institutional Effectiveness (2012) Small office focused on compliance reporting Colleague ERP system Operational Data Store (ODS) acquired 2010 No reporting tools, ODS used by Institutional

Research staff only Limited access to data, no agreed-upon success

metrics

Starting Point - 2012

Many emailed spreadsheets, some reports available on the intranet, many different ways to query for similar information, lots of requesting and waiting for data.

Phase 1, Data/Social: 2012-2013

2012 2013• Data needs• Achieving the Dream (AtD)• Purchased Data Warehouse • Developed initial student

success metrics based on Achieving the Dream cohorts

• Report development

• Data Warehouse installation• Scorecard• Regularized reporting• SharePoint Business

Intelligence

Why SharePoint?

Familiar interface Familiar tools (Excel) Completely customizable Flexibility – can deliver many types of

documents

Connection with IT is vital!

IT – SharePoint administration– Data connections– Data modeling– Database creation and maintenance

IE – Query building in SQL and visual tools– Users needs– Report development

2014 2015• Achievement Analytics • Automation • Roll-out and training • Data Governance

• AtD Leader College• 200+ Achievement Analytics

users• AACC Pathways Project • Next steps - predictive analytics

Phase 2, Display: 2014-2017

Phase 2 - display

Centralizing data into SharePoint sources– Data models (Powerpivot, Power query)– SSRS (SQL Server Reporting Services)– Power BI

Transition from manual process – Course Section Reports – had required 15 - 20

hours per week to create paper copies -- now automatic

Homepage

Student Success Section

Course Success Report

Student Initiatives Challenge:

Large amount of initiatives (50+)

No overarching goals, siloes

Each assessment has different outcomes, logic models or not assessment

Assessment: Getting knowledge of each initiative (from each

initiative leader)

Gathering the data (different systems, different programs, different formats)

Create an evaluation/assessment plan with common outcomes

Student Success Evaluations

Student Success: Evaluation PlanIntervention: Early Alert System, FOCUS Evaluation Model

Outputs Satisfaction Outcomes Critical & Measurable

Observable Behaviors Outcomes Success Metrics Outcomes

Number of courses using the Early Alert system (active users: faculty responding surveys and/or raising notices in the system). Number of students in courses that use the EAS. Percentage of students that received:

1) An academic performance notice

2) Tutoring referral 3) Kudos 4) Other notice

Professional Development training (Focus on FOCUS, and Fully FOCUSed)

Early Alert students survey:

1) Awareness 2) Actions after notice 3) Contribution of the system

to success completion (perception)

4) System problems Early Alert faculty survey :

1) Usage of system 2) Feedback of system 3) Opportunities of

improvement Training surveys:

1) Knowledge of system 2) Intended to use the system 3) Recommendation

Number of students and percentages:

1) Contact retention specialist 2) Attend tutoring 3) Improve grade 4) Meet with advisor 5) Complete missing academic

work 6) Improve attendance 7) Improve participation 8) Approach faculty

Number and percentages:

1) Successful completion (department, modality, and targeted courses)

2) Withdrawal (department, modality, and targeted courses)

3) Retain (Autumn to Spring, and Autumn to Autumn).

4) Persistence in their pathway.

Assumption(s): EAS notices, nudges and personal contacts with students, particularly those at risk, will result in students taking corrective action to improve success.

Assessment of Quality of Effort: Degree of faculty involvement in designing the system, level of training in how to use the system, use of best practices, shared faculty & staff responsibility in following up with students that receive notices, but do not immediately respond, % of classes and students covered by EAS.

Comparison Groups: Students in courses where instructors are not using the Early Alert System, during the term.

Student Success: Evaluation Plan

Student Success: Evaluation Data

More Participation = Better Results? Assumption: participation contributes to students’

success

Outcomes: course success, GPA, retention

Identify the students participating in the different initiatives

Clustering (by number of initiatives that impact each student)

Multiple Success Initiatives:1) Orientation

2) Placement Boot Camp

3) COLS-1100 /1101(College Success course)

4) Developmental Education Initiatives (English)

5) Developmental Education Initiatives (Math)

6) Service Learning

7) Learning Communities

Some initiatives embedded in class /course design (students do not know they are participating)

Multiple Success Initiatives:8) Generation One Trailblazers

9) Honors Program

10) Student Support Services / First-Year Experience (TRIO)

11) Out of Class Academic Support Services

12) Early Alert

13) Blueprint Workshops (advising)

Some target specific populations

Some require self-selection

Objective Population: New to CSCC

Not transient

Exclude high school / dual enrollment

– Autumn 2014: 5,513 new students



Distribution of Students by Number of Initiatives

4%

35%

37%

19%

4%

1%

4%

32% 33%

22%

8%

1%

6%

31%

26%

23%

11%

3%

0%

5%

10%

15%

20%

25%

30%

35%

40%

None One Two Three Four Five or More

Autumn 2014 Autumn 2015 Autumn 2016

“Zero Initiative” Students

28% of these students dropped before the third week of the semester (earn no credits) AU14. This percentage increase to 38% in AU16

During AU14, 39% of the grades obtained corresponded to Skilled Trade courses (APPL). In AU16, 55% of their grades corresponded to Skilled Trade courses (APPL)

Some students have already earned a prior degree (bachelors and /or master’s)

Gender by Number of Initiatives (14AU)

50% 49%

56%

65% 64%

50% 51%

44%

35% 36%

0%

10%

20%

30%

40%

50%

60%

70%

One Two Three Four Five or More

Female Male

Gender by Number of Initiatives (16AU)

49%

53% 53%

60% 61%

51%

47% 47%

40% 39%

0%

10%

20%

30%

40%

50%

60%

70%


Female Male

Race/Ethnicity by Number of Initiatives (14AU)

19% 19%

24%

28%

45%

18% 18% 19%20%

18%

63% 63%

57%

52%

36%

0%

10%

20%

30%

40%

50%

60%

70%


BL Other WH

Race/Ethnicity by Number of Initiatives (16AU)

17%

22% 22%24%

29%

25%

22% 22% 23%

28%

58%56% 56%

53%

43%

0%

10%

20%

30%

40%

50%

60%

70%


BL Other WH

Academic Status by Number of Initiatives (14AU)

27%

48%

60%

70%

87%

73%

52%

40%

30%

13%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Full-time Part-time

Academic Status by Number of Initiatives (16AU)

31%

47%

63%

71%

64%

69%

53%

37%

29%

37%

0%

10%

20%

30%

40%

50%

60%

70%

80%


Full-time Part-time

Pell-Eligibility by Number of Initiatives (14AU)

38%

46%

51%

57%

67%

62%

54%

49%

43%

33%

0%

10%

20%

30%

40%

50%

60%

70%

80%


Yes No

Pell-Eligibility by Number of Initiatives (16AU)

32%

48%

53%

59%

65%68%

52%

47%

41%

35%

0%

10%

20%

30%

40%

50%

60%

70%

80%


Yes No

Average GPA by Number of Initiatives

2.092.03

2.25

2.55

2.28

2.142.04

2.28

2.00

2.62

2.18

2.05

2.33

2.552.46

0.00

0.50

1.00

1.50

2.00

2.50

3.00


Autumn 2014 Autumn 2015 Autumn 2016

Course Success by Number of Initiatives

63%66%

73%

81%

75%

62%64%

73%

78%81%

65% 64%

73%

79%77%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

One Two Three Four Five or MoreSucc AU14 Succ AU15 Succ AU16

Retention by Number of Initiatives (14AU)

58%

67%

76%

86%

91%

37%

48%

56%

67%71%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Retention AU14-Sp15 Retention AU14- AU15

Retention by Number of Initiatives (16AU)

61%

71%

80%

86%

92%

39%

48%

58%

64% 64%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%


Retention AU16-SP17 Retention AU16-AU17

Tying It All Together

“Small-scale” initiative assessment data reinforces and confirms annual cohort-level course success data

Success interventions, when added together, make an impact on overall success rates

“Big Picture” data confirmed

AtD Cohort Course Success: Annual Improvement

62.8% 64.2%66.6% 67.7%

65.9%68.5%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

2011/12 2012/13 2013/14 2014/15 2015/16 2016/17

AtD Success by Gender: Trending Upward

64.0%67.2% 68.0%

69.6%67.7%

71.0%

61.3% 61.3%

65.1% 65.5% 64.1%65.7%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

2011/12 2012/13 2013/14 2014/15 2015/16 2016/17

Female Male

AtD Success by Race/Ethnicity: Success Gap Reduced

47.4%

52.2% 52.8%55.5% 55.0%

56.7%

69.4% 68.5%70.5%

72.0%69.5%

72.4%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

2011/12 2012/13 2013/14 2014/15 2015/16 2016/17

Black or African-American White

AtD Success by Pell-Eligibility: Success Gap Reduced by 50%

55.2%

58.8%61.2%

62.6% 61.6%64.0%

73.3%69.7%

71.3% 72.1%69.9%

72.4%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

2011/12 2012/13 2013/14 2014/15 2015/16 2016/17

Pell Non-Pell

Challenges Data collection

– Data in different systems or manually compiled

– More initiatives to be included

– Some students are not impacted by any initiative and dropped all their classes during the first three weeks

Success Initiatives to be included:

1) Connect2Complete Peer Advocate Program

2) AmeriCorps College Completion Coach

3) Global Diversity and Inclusion

4) IPAD initiative

5) Career Counseling for Undeclared Students

6) Student Financial Literacy Project

7) Student Engagement and Leadership (OrgSync Platform)

8) Moving to Scale with Active Learning

Questions ??

Thank You for Attending

Julio E. [email protected]

614-287-2337

mailto:[email protected]

Assessing Student Success Initiatives · 2017. 11. 15. · 2015 • Achievement Analytics •...

Documents

Transcript of Assessing Student Success Initiatives · 2017. 11. 15. · 2015 • Achievement Analytics •...