THE ROLE OF CALIBRATION IN ADVANCING FACULTY LEARNING

ABOUT STUDENT LEARNING

Terry Rhodes & Ashley Finley

AAC&U

Institute on Integrative Learning and the Departments

Portland, OR

“OUR” COMMON GROUND:

Criteria

The Anatomy of a VALUE RubricThe Anatomy of a VALUE Rubric

Levels

Performance Descriptors

THE CALIBRATION TRAINING PROCESS Scoring Steps:

Review rubric to familiarize yourself with structure, language, performance levels

Ask questions about the rubric for clarification or to get input from others regarding interpretation

Read student work sample Connect specific points of evidence in work sample

with each criterion at the appropriate performance level (if applicable)

Calibration Steps: Review scores Determine common score(s) Hear from outliers Discuss Determine final score

THE GROUND RULES

This is not grading. Think globally about student work and about

the learning skill. Think beyond specific disciplinary lenses or content.

We are not changing the rubric (today). Our work is time sensitive. Go with your

instinct. Start with 4 and work backwards. Pick one

performance benchmark per criterion. Avoid “.5”.

Zero does exist. Assign “0” if work does not meet benchmark (cell one) performance level. N/A exists. Assign “not applicable” if the student work is not intended to meet a particular criterion.

SIGNATURE ASSIGNMENTS Assignment should enable attainment of

criteria Break down criteria to determine key components for

assignment

What should students do with content to meet criteria? E.g. What are the pieces to be analyzed, compared,

integrated?

Will the assignment be used for more than one outcome?

What are the types of assignments that will be most helpful for allowing students to demonstrate competency?

EXAMPLE OF PROCESS

Step 1: All Gen Ed Courses reported as addressing and assessing Info. Tech. Literacy identified as potential courses from which to request artifacts. (54 courses)

Step 2: Of courses identified, approx. 20% were randomly selected for sample (10 courses, 36 total sections)

Step 5: Artifacts submitted to Director of Learning Outcomes for scoring. (66 artifacts)

Step 4: Start of semester, department chairs notified of courses in from which artifacts were to be requested. Chairs worked with individual faculty to fulfill request.

Step 3: Within each selected course, 2 students randomly selected by roster # to submit artifacts (74 artifacts)

Step 6: Faculty scoring team met at the close of spring semester for a norming session and scoring. (62 artifacts)

From: Carroll Community College

Flow chart of sequential steps in the request, submission, and scoring of student artifacts for Learning Goal 4: Information and technology literacy.

CAMPUS EXAMPLE OF OUTCOMES ASSESSMENT USING RUBRIC DATA

Dimension % of students who scored 2 or higher

% of students who scored 3 of higher

Explanation of Issues 68.3 35.5

Interpreting & Analysis

65.0 28.2

Influence of Context and Assumptions

48.8 21.2

Student’s position 54.5 24.0

Conclusions and related outcomes

47.7 17.0

From: UNC-Wilmington, Critical Thinking Rubric

BUILDING THE EVIDENTIARY BASE: UNIVERSITY OF KANSAS

Per

cen

t o

f R

atin

gs

Critical Thinking: Issues, Analysis, and Conclusions

Inter-rater reliability = >.8

BUILDING THE EVIDENTIARY BASE: UNIVERSITY OF KANSAS

Per

cen

t o

f R

atin

gs

Critical Thinking: Evaluation of Sources and Evidence

BUILDING THE EVIDENTIARY BASE: UNIVERSITY OF KANSAS

Per

cen

t o

f R

atin

gs

“VALUE added” for 4 years - writing