2014 AIR Reporting Program-level Retention and Graduation
22
Reporting Program-Level Retention and Graduation Alison Joseph & David Onder AIR 2014
-
Upload
david-onder -
Category
Education
-
view
48 -
download
1
Transcript of 2014 AIR Reporting Program-level Retention and Graduation
Reporting Program-Level Retention
and Graduation
Alison Joseph & David Onder
AIR 2014
• 10,107 students
• Master’s Comprehensive
• Mountain location
• Residential and Distance
2
The Problem
• We are asked for program-level data
– Lack of program-level retention and graduation rates
– Complicated (particularly for undergrads)
– Different programs serve different purposes
– High stakes – program prioritization (AA driven)
• Reports lumped all non-retained together (whether
they graduated or stopped out)
3
Background
• Historically reported on freshmen cohort
– University-level only
– Only one segment of our population
• No info on grad students
• No info on transfers
• No info on those who start part-time
4
What We Wanted
• Solid & simple approach (easy to explain and defend)
• Fair
– Useful for all types of programs
• Meaningful for decision-making (high- and low-level)
• Not overly complicated display
• Illuminates
– Overall performance
– Historic trends
– When are students lost
• Something that can be generated yearly w/o too much
effort
5
5 Outcomes
• Five possible outcomes for each student that declares
a given major
– Retained in program
– Graduated in program
– Retained in different program
– Graduated in different program
– Not retained (stop-out/drop-out)
• Exclusive and exhaustive
6
General Approach
• Based on cohorts:
– A student is placed in a program cohort the 1st time they
declare a given program
– Each student in the cohort is flagged as one of the 5 possible
outcomes for each ½ year interval (each regular semester)
– At each interval we report where the members of the cohort
fall
– Each student will only appear in one cohort for a program
(usually)
7
Why this works
• Students are not double-counted (usually)
• We can report data on any interval, if asked
• If a student stops-out, then returns, they are picked
back up
• Bridges the gap between retention reports and
graduation reports
8
Technical Approach
• Used SAS to generate data set
• Data put in Excel sheet
• There is one report built per level
– Drop down lists shows all the programs at that level
– Formulas reference program code, and populate report based
on that code (sumifs, countifs, averageifs, etc.)
– Expanded on techniques used for our Fact Book automation
9
Multiple Iterations
Started with term-level data
New Cohort@1 year @2 years @3 years @4 years @6 years
0.5 1 2 3 4 6
Year SemesterNew Cohort Prg Ret Prg Grad WCU Ret WCU Grad
Not Retained
Total Prg Ret Prg Grad WCU Ret WCU GradNot
RetainedTotal Prg Ret Prg Grad WCU Ret WCU Grad
Not Retained
Total Prg Ret Prg Grad WCU Ret WCU GradNot
RetainedTotal Prg Ret Prg Grad WCU Ret WCU Grad
Not Retained
Total
1996 Fall 28 54% 18% 4% 0% 25% 100% 25% 46% 0% 0% 29% 100% 21% 61% 0% 0% 18% 100% 14% 64% 0% 0% 21% 100% 7% 79% 0% 0% 14% 100%
1997 Spring 3 67% 0% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100% 0% 33% 0% 0% 67% 100% 0% 33% 0% 0% 67% 100%
1997 Fall 14 50% 14% 0% 0% 36% 100% 7% 43% 0% 0% 50% 100% 14% 50% 0% 0% 36% 100% 0% 50% 0% 0% 50% 100% 0% 57% 0% 0% 43% 100%
1998 Spring 4 75% 0% 0% 0% 25% 100% 25% 50% 0% 0% 25% 100% 0% 75% 0% 0% 25% 100% 0% 75% 0% 0% 25% 100% 0% 75% 0% 0% 25% 100%
1998 Fall 15 80% 7% 0% 0% 13% 100% 47% 40% 0% 0% 13% 100% 13% 53% 0% 0% 33% 100% 13% 60% 0% 0% 27% 100% 7% 67% 0% 0% 27% 100%
1999 Spring 5 60% 0% 20% 0% 20% 100% 40% 0% 0% 20% 40% 100% 0% 40% 0% 20% 40% 100% 0% 40% 0% 20% 40% 100% 0% 40% 0% 20% 40% 100%
1999 Fall 4 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100%
2000 Spring 4 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100% 25% 25% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100%
2000 Fall 3 67% 0% 0% 0% 33% 100% 0% 0% 0% 0% 100% 100% 33% 0% 0% 0% 67% 100% 0% 33% 0% 0% 67% 100% 0% 33% 0% 0% 67% 100%
2001 Spring 5 40% 0% 0% 0% 60% 100% 20% 20% 0% 0% 60% 100% 20% 20% 0% 0% 60% 100% 0% 40% 0% 0% 60% 100% 0% 40% 0% 0% 60% 100%
2001 Fall 9 56% 22% 11% 0% 11% 100% 22% 56% 11% 0% 11% 100% 22% 56% 0% 11% 11% 100% 0% 78% 0% 11% 11% 100% 0% 78% 0% 11% 11% 100%
2002 Spring 5 60% 0% 0% 0% 40% 100% 60% 0% 0% 0% 40% 100% 0% 0% 0% 0% 100% 100% 0% 0% 0% 0% 100% 100% 20% 0% 0% 0% 80% 100%
2002 Fall 10 60% 10% 10% 0% 20% 100% 20% 40% 0% 10% 30% 100% 10% 50% 10% 10% 20% 100% 10% 50% 0% 10% 30% 100% 0% 60% 0% 10% 30% 100%
2003 Spring 6 67% 0% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100% 17% 50% 0% 0% 33% 100% 0% 67% 0% 0% 33% 100% 17% 67% 0% 0% 17% 100%
2003 Fall 11 73% 9% 0% 0% 18% 100% 36% 45% 0% 0% 18% 100% 9% 73% 0% 0% 18% 100% 0% 73% 9% 0% 18% 100% 0% 73% 0% 9% 18% 100%
2004 Spring 4 75% 0% 0% 0% 25% 100% 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100% 25% 25% 0% 0% 50% 100% 0% 25% 0% 0% 75% 100%
2004 Fall 12 67% 0% 17% 0% 17% 100% 25% 42% 0% 8% 25% 100% 0% 50% 0% 8% 42% 100% 0% 50% 0% 8% 42% 100% 0% 50% 0% 8% 42% 100%
2005 Spring 3 67% 0% 0% 0% 33% 100% 67% 0% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100% 0% 67% 0% 0% 33% 100% 0% 67% 0% 0% 33% 100%
2005 Fall 17 71% 6% 0% 0% 24% 100% 35% 41% 0% 0% 24% 100% 12% 53% 6% 0% 29% 100% 6% 59% 0% 6% 29% 100% 0% 65% 0% 6% 29% 100%
2006 Spring 8 50% 13% 0% 13% 25% 100% 25% 13% 25% 13% 25% 100% 13% 25% 13% 25% 25% 100% 0% 38% 0% 38% 25% 100% 0% 38% 0% 38% 25% 100%
2006 Fall 7 57% 0% 0% 0% 43% 100% 0% 29% 29% 0% 43% 100% 0% 29% 14% 14% 43% 100% 0% 29% 0% 14% 57% 100% 0% 0% 0% 0% 0% 0%
2007 Spring 9 33% 22% 22% 0% 22% 100% 0% 56% 22% 0% 22% 100% 0% 56% 0% 11% 33% 100% 0% 56% 0% 11% 33% 100% 0% 0% 0% 0% 0% 0%
2007 Fall 13 92% 0% 0% 0% 8% 100% 38% 46% 0% 0% 15% 100% 15% 69% 0% 0% 15% 100% 8% 77% 0% 0% 15% 100% 0% 0% 0% 0% 0% 0%
2008 Spring 3 33% 0% 0% 0% 67% 100% 0% 33% 0% 0% 67% 100% 0% 33% 33% 0% 33% 100% 0% 33% 0% 0% 67% 100% 0% 0% 0% 0% 0% 0%
2008 Fall 18 67% 6% 6% 0% 22% 100% 22% 50% 6% 0% 22% 100% 17% 61% 6% 0% 17% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2009 Spring 7 57% 0% 0% 0% 43% 100% 57% 14% 0% 0% 29% 100% 29% 43% 0% 0% 29% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2009 Fall 15 80% 0% 7% 0% 13% 100% 27% 53% 0% 7% 13% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2010 Spring 3 67% 0% 0% 0% 33% 100% 67% 0% 0% 0% 33% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2010 Fall 17 76% 0% 0% 0% 24% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2011 Spring 6 50% 0% 0% 0% 50% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2011 Fall 9 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
New Cohort@1 year @2 years
0.5 1 2
Year SemesterNew Cohort Prg Ret Prg Grad WCU Ret WCU Grad
Not Retained
Total Prg Ret Prg Grad WCU Ret WCU GradNot
RetainedTotal
1996 Fall 28 54% 18% 4% 0% 25% 100% 25% 46% 0% 0% 29% 100%
1997 Spring 3 67% 0% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100%
1997 Fall 14 50% 14% 0% 0% 36% 100% 7% 43% 0% 0% 50% 100%
1998 Spring 4 75% 0% 0% 0% 25% 100% 25% 50% 0% 0% 25% 100%
1998 Fall 15 80% 7% 0% 0% 13% 100% 47% 40% 0% 0% 13% 100%
1999 Spring 5 60% 0% 20% 0% 20% 100% 40% 0% 0% 20% 40% 100%
1999 Fall 4 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100%
2000 Spring 4 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100%
10
Academic Year
Next combined data into academic years
New Cohort@1 year @2 years
0.5 1 2
YearNew Cohort Prg Ret Prg Grad WCU Ret WCU Grad
Not Retained
Total Prg Ret Prg Grad WCU Ret WCU GradNot
RetainedTotal
2003-2004 15 73% 7% 0% 0% 20% 100% 47% 33% 0% 0% 20% 100%
2004-2005 15 67% 0% 13% 0% 20% 100% 33% 33% 0% 7% 27% 100%
2005-2006 25 64% 8% 0% 4% 24% 100% 32% 32% 8% 4% 24% 100%
2006-2007 16 44% 13% 13% 0% 31% 100% 0% 44% 25% 0% 31% 100%
2007-2008 16 81% 0% 0% 0% 19% 100% 31% 44% 0% 0% 25% 100%
2008-2009 25 64% 4% 4% 0% 28% 100% 32% 40% 4% 0% 24% 100%
2009-2010 18 78% 0% 6% 0% 17% 100% 33% 44% 0% 6% 17% 100%
2010-2011 23 70% 0% 0% 0% 30% 100%
11
Summary
@1 year @2 years
Year New Cohort Program Success Non-program Success Not Retained Program Success Non-program Success Not Retained
2003-2004 15 80% 0% 20% 80% 0% 20%
2004-2005 15 67% 13% 20% 67% 7% 27%
2005-2006 25 72% 4% 24% 64% 12% 24%
2006-2007 16 56% 13% 31% 44% 25% 31%
2007-2008 16 81% 0% 19% 75% 0% 25%
2008-2009 25 68% 4% 28% 72% 4% 24%
2009-2010 18 78% 6% 17% 78% 6% 17%
2010-2011 23 70% 0% 30%
All students lumped into 3 groups
This is the most summarized data we can (read: are willing to) provide.
The bottom line = Bold 3-group number
12
But how does that compare?
@1 year @2 years
Year New Cohort Program Success Non-program Success Not Retained Program Success Non-program Success Not Retained
2003-2004 10.74 72% 4% 24% 65% 7% 28%
2004-2005 12.52 65% 8% 27% 58% 11% 32%
2005-2006 12.33 51% 20% 29% 46% 20% 34%
2006-2007 12.73 62% 13% 26% 58% 12% 30%
2007-2008 12.53 73% 3% 24% 68% 2% 30%
2008-2009 12.92 72% 2% 26% 68% 3% 29%
2009-2010 13.53 74% 1% 25% 63% 4% 34%
2010-2011 13.70 75% 1% 24%
This is average program data to use as a comparison
13
Visual Representation
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0.5 1
1.5 2
2.5 3
3.5 4
4.5 5
5.5 6
6.5 7
7.5 8
8.5 9
9.5 10
NR
WCU G
Pgm G
WCU R
Pgm R0
5
10
15
20
25
30
35
40
45
50
0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5
Prg R
WCU R
NR
Pgm G
WCU G
0
5
10
15
20
25
30
35
40
45
50
0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5
Pgm R
Pgm G
All R
All G
14
Horizontal stacked bar
• Normally we would never do this, but …..
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
Program Success Non-program Success Not Retained
What are people asking:
- How is the program performing?
- How are students performing overall at institution?
- How many are dropping out?
15
We graph 5 Flags too
0% 50% 100%
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
@ 1 year
0% 50% 100%0% 50% 100% 0% 50% 100% 0% 50% 100%
@ 2 year @ 3 year @ 4 year @ 6 year
This works because:
Horizontally
- See people transitioning to completers and drop-outs over time
- Follow specific cohorts
Vertically
- Compare performance of different cohorts at the same intervals
16
Why this report is REALLY awesome
• Use a special approach with a named range to find a
list of all unique programs, and populate drop-down
box with this list
(http://www.datawright.com.au/excel_resources/excel_
dynamic_ranges.htm)
• VBA to cycle through, do calculations, print PDF out
to a directory (by department and college) and move
on to next report
17
Expanding the Idea
• Changing the reporting level
– CIP code (groups up old and new program codes)
– Department (helpful if similar programs that students transfer
between)
– College
18
Ideas for Next Steps
• True Success Rate (VSA, incorporating of
Clearinghouse Data)
• Consider rolling averages or other approaches to
smooth turbulent data on small groups
• Compare retention data against unit-level goals
19
Contact Information
David Onder, Director of Assessment
Alison Joseph, Applications Analyst
Office of Institutional Planning and Effectiveness
oipe.wcu.edu, (828) 227-7239
20
Reporting Program‐Level Student Success Measures AIR 2014
Alison Joseph & David Onder
• Problem ‐ We are asked for program‐level success data (retention and graduation) • Complicated (particularly for undergrads) • Different programs serve different purposes • High stakes – program prioritization (AA driven)
– Reports lumped all non‐retained together (whether they graduated or stopped out) • Background
– Historically reported on freshmen cohort, University‐level only – Only one segment of our population (no graduate students, no transfers, no part‐time)
• What We Wanted – Solid & simple approach (easy to explain and defend) – Fair (Useful for all types of programs) – Meaningful for decision‐making (high‐ and low‐level) – Not overly complicated display – Illuminates: overall performance, historic trends, when students are lost – Something that can be generated yearly w/o too much effort
• 5 Outcomes – Five outcomes for each student that declares a given major: Retained in program, Graduated in program, Retained in different
program, Graduated in different program, Not retained (stop‐out/drop‐out) ‐ Exclusive and exhaustive • General Approach (based on cohorts)
– A student is placed in a program cohort the 1st time they declare a given program – Each student is flagged as one of the 5 possible outcomes for each ½ yr interval (each regular semester) – At each interval we report where the members of the cohort fall – Each student will only appear in one cohort for a program (usually)
• Why this works – We can report data on any interval, if asked – If a student stops‐out, then returns, they are picked back up – Bridges the gap between retention reports and graduation reports
• Technical Approach – Used SAS to generate data set; Excel for reporting (Expanded on techniques used for our Fact Book automation) – There is one report built per level
• Drop down lists shows all the programs at that level • Formulas reference program code, and populate report based on that code (sumifs, averageifs, etc.)
• Horizontal stacked bar
• We graph 5 Flags too
– Horizontally ‐ See people transitioning to completers and drop‐outs over time, follow specific cohorts – Vertically ‐ Compare performance of different cohorts at the same intervals
• Why this report is REALLY awesome – Use a special approach with a named range to find a list of all unique programs, and populate drop‐down box with this list
(http://www.datawright.com.au/excel_resources/excel_dynamic_ranges.htm) – VBA to cycle through, do calcs, print PDF out to a directory (by dept and college) and move on to next report
• Expanding the Idea – CIP code (groups up old and new program codes), Dept (similar programs that students transfer between), College
• Ideas for Next Steps – True Success Rate (VSA, incorporating of Clearinghouse Data) – Consider rolling averages or other approaches to smooth turbulent data on small groups – Compare retention data against unit‐level goals
• Contact Information – Alison Joseph, Applications Analyst ([email protected]) – David Onder, Director of Assessment ([email protected])
– Office of Institutional Planning and Effectiveness ‐ oipe.wcu.edu , (828) 227‐7239
0% 20% 40% 60% 80% 100%
2003‐2004
2005‐2006
2007‐2008
2009‐2010
Program Success Non‐program Success Not Retained
What are people asking?• How is the program performing? • How are students performing overall at
institution? • How many are dropping out?
Program
CIP (12‐digit):
910000000000
Program
Nam
eInform
ation M
anagement ‐ BS
Career Level:
Undergraduate
Degree Level:
Bachelors
Program CIP (12‐digit):
910000000000
Program Department:
Inform
ation M
anagement
Program College:
College
of Inform
ation Studies
New
Cohort
0.5
12
46
Year
New Cohort
CIP Ret
CIP Grad
WCU Ret
WCU
Grad
Not
Retained
Total
CIP Ret
CIP Grad
WCU Ret
WCU
Grad
Not
Retained
Total
CIP Ret
CIP Grad
WCU Ret
WCU
Grad
Not
Retained
Total
CIP Ret
CIP Grad
WCU Ret
WCU
Grad
Not
Retained
Total
2005‐2006
13
69%
0%
8%
0%
23%
100%
54%
0%
23%
0%
23%
100%
0%
46%
31%
0%
23%
100%
0%
46%
0%
15%
38%
100%
2006‐2007
875%
0%
13%
0%
13%
100%
38%
13%
13%
0%
38%
100%
0%
38%
13%
0%
50%
100%
0%
38%
0%
13%
50%
100%
2007‐2008
15
73%
0%
13%
0%
13%
100%
40%
0%
40%
0%
20%
100%
0%
33%
7%
13%
47%
100%
0%
33%
7%
20%
40%
100%
2008‐2009
13
54%
0%
31%
0%
15%
100%
38%
8%
31%
0%
23%
100%
15%
31%
8%
8%
38%
100%
2009‐2010
23
30%
9%
22%
4%
35%
100%
17%
9%
9%
4%
61%
100%
4%
13%
9%
13%
61%
100%
2010‐2011
18
50%
0%
17%
0%
33%
100%
33%
11%
28%
0%
28%
100%
2011‐2012
23
65%
0%
9%
0%
26%
100%
39%
0%
30%
0%
30%
100%
2012‐2013
15
53%
0%
0%
0%
47%
100%
University Average
Year
New Cohort
CIP Ret
CIP Grad
WCU Ret
WCU
Grad
Not
Retained
Total
CIP Ret
CIP Grad
WCU Ret
WCU
Grad
Not
Retained
Total
CIP Ret
CIP Grad
WCU Ret
WCU
Grad
Not
Retained
Total
CIP Ret
CIP Grad
WCU Ret
WCU
Grad
Not
Retained
Total
2005‐2006
12.07
62%
2%
15%
1%
20%
100%
39%
13%
18%
2%
28%
99%
4%
44%
5%
14%
31%
99%
0%
48%
0%
19%
32%
99%
2006‐2007
12.67
59%
2%
14%
1%
25%
100%
37%
11%
17%
1%
34%
101%
5%
38%
8%
12%
37%
101%
1%
43%
1%
18%
38%
101%
2007‐2008
14.05
58%
3%
15%
1%
23%
100%
37%
12%
18%
2%
31%
100%
6%
39%
6%
12%
36%
100%
0%
45%
1%
16%
38%
100%
2008‐2009
13.08
55%
4%
15%
1%
23%
100%
34%
16%
17%
4%
29%
100%
4%
42%
5%
9%
39%
100%
2009‐2010
13.06
58%
3%
13%
1%
25%
100%
37%
17%
10%
2%
34%
100%
5%
45%
5%
9%
37%
100%
2010‐2011
12.47
60%
4%
8%
0%
27%
100%
38%
20%
12%
1%
29%
100%
2011‐2012
14.36
61%
4%
9%
0%
25%
100%
37%
18%
13%
1%
31%
100%
2012‐2013
14.76
59%
4%
14%
0%
22%
100%
* 12‐Digit CIP codes are used in
the North Carolina state system. The first 6 digits are used for the discipline, the next three for the degree code, and the final three for the specialty code.
Retention and Graduation ‐ by 12‐digit CIP Code
CIP Code: 910000000000
@6 years
@1 year
@2 years
@4 years
@6 years
@1 year
@2 years
@4 years
0%
20%
40%
60%
80%
100%
Program
Retention
Program
Graduation
WCU Retention
WCU Graduation
Not Retained
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Program
Retention
Program
Graduation
WCU Retention
WCU Graduation
Not Retained
Legend
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
Legend
