Why Assessment Results Are Hard to Use
(and what to do about it)
Hi…I’m the Assessment Director
The Theory
What Could Go Wrong?
Look familiar?
Assessment Design
Choosing an Achievement Scale
Pick a scale that corresponds to the educational progress we expect to see.
Example:
0 = Remedial work
1 = Fresh / Soph level work
2 = Jr / Sr level work
3 = What we expect of our graduates
Subjective measurements
Use authentic data that already exists Sacrifices reliability for validity
Create subjective ratings in context Sacrifices reliability for validity
Assess what you observe, not just what you teach! Recovers some reliability
For complex goals like general education:
www.coker.edu/assessment
Survey Attitudes and Behaviors
CIRP, BCSSE, Student satisfaction inventory Link student ID numbers to other indicators These can be easily and productively
outsourced
Analysis and Reporting
Some Tools
ANOVA Logistic Regression and ROC curves
http://statpages.org/logistic.html Pivot tables
Longitudinal Analysis
Use data you already have
Grades FAFSA, SAT, ACT items Demographic data ePortfolio statistics or work Library circulation statistics
Analysis: Example
Writing by Class and Library Use
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Fall 05 Fall 04 Fall 03 Fall 02
Start Term
Avg
. W
riti
ng
Sco
re
> Library
< Library
You don’t have to average
Measurement requires: Units Ability to aggregate
What we actually do is estimation Without measurement we shouldn’t average
Averages
Proportions
Min / max
Major Code Class Student ID Last First ObservationsArt (4) Senior xxxxxxx Tatiana Tolstoy 7
Min Max AvgAnalytical Thinking Remedial Jr/Sr 1.00Creative Thinking Remedial Fr/So 0.80Writing Effectiveness Fr/So Jr/Sr 1.17Speaking Effectiveness Remedial Jr/Sr 1.00
Confidence Intervals
Longitudinal Writing Effectiveness in Day
0.00
0.50
1.00
1.50
2.00
2.50
2003-4 2004-5 2005-6 2006-7
Academic Year
Avg
. S
core AF F
AF M
W F
W M
Portfolio analysis: epic FAIL!
Grammar Scores
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
07 06 05 04
Entering Year
Ave
rag
e S
core
Day
Evening
Analysis: Example
Proportions
Day Analytical Min Rating
0%
20%
40%
60%
80%
100%
2003-4 2004-5 2005-6 2006-7
Jr-Sr
Fr-Soph
Remedial
Conclusions
Design it yourself Use a sensible longitudinal scale Combine with other data Avoid averages
Last Requests?
highered.blogspot.com
Top Related