Putting Learning Reconsidered Into Practice: Developing...
Transcript of Putting Learning Reconsidered Into Practice: Developing...
![Page 1: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/1.jpg)
© 2007 Keeling & Associates, LLC
Putting Learning Reconsidered Into Practice: Developing and Assessing Student Learning Outcomes
Learning Reconsidered InstituteSt. Louis, MOJune 6, 2007
Richard P. Keeling, MDRic Underhile, PhD
![Page 2: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/2.jpg)
© 2007 Keeling & Associates, LLC
Learning, assessment, and the changing mandate of Student Affairs
• Responding to a revised, broader understanding of learning that integrates development
• Mapping and supporting learning inside and outside classrooms
• Engagement of students in active, transformative learning
• Leadership in making the whole campus a learning community and in enhancing student experience and student success
• Identifying and ameliorating barriers to learning
• Developing, implementing, and assessing student learning outcomes
2
![Page 3: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/3.jpg)
© 2007 Keeling & Associates, LLC
What are learning outcomes?
3
• Learning outcomes define the goals of learning experiences (activities, courses, majors, curricula, engagement with post-secondary education)
• What any student should be able to do, know, or value as a result of engaging in that learning experience
• Not the same as satisfaction with education, professors, facilities, activities, or experiences; not the same as outputs, such as grades, GPA, retention rate, graduation rate
• Outcomes define impact -- how the student has changed.
• The achievement of learning outcomes (student success) measures institutional effectiveness.
![Page 4: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/4.jpg)
© 2007 Keeling & Associates, LLC
Outcomes-based thinking shifts and shares our attention
Process Outcome
Planning activities and things to “do”
How the student will change as a result of what we do
“What are we going to do?”
Counting and asking
“How will students be different; how will they learn, change, grow?”
Documenting change; showing results
4
![Page 5: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/5.jpg)
© 2007 Keeling & Associates, LLC
Learning how to develop and assess student learning outcomes
• New language and concepts
• Collaborative work process
• Comfort with critique, data, and feedback
• Reframing of assessment thinking from process and procedural metrics
• Realignment of concepts from assessing students to assessing institutional effectiveness
• Departure from testing paradigm
• Cross-fertilization5
![Page 6: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/6.jpg)
© 2007 Keeling & Associates, LLC
Learning to develop, implement, and assess student learning outcomes
6
Curricular
Collaborative
Asset-based
Practice
Mistakes OK
Parsimonious
Iterative
![Page 7: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/7.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
Purposes
7
![Page 8: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/8.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
7
![Page 9: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/9.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
Linkages
Look upLook downLook around
7
![Page 10: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/10.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
Linkages
Look upLook downLook aroundLook aroundLook down
Look upMissionContext
Colleagues
7
![Page 11: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/11.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
Linkages
Look upLook downLook aroundLook aroundLook down
Look upMissionContext
Colleagues
Priorities
Don’t paint learning
outcomes on
programs
7
![Page 12: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/12.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
Linkages
Look upLook downLook aroundLook aroundLook down
Look upMissionContext
Colleagues
Priorities
Don’t paint learning
outcomes on
programs
Review programs, services, activities
7
![Page 13: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/13.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
Linkages
Look upLook downLook aroundLook aroundLook down
Look upMissionContext
Colleagues
Priorities
Don’t paint learning
outcomes on
programs
Review programs, services, activities
Practice
SLOs
MetricsAssessment Plans
7
![Page 14: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/14.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
Linkages
Look upLook downLook aroundLook aroundLook down
Look upMissionContext
Colleagues
Priorities
Don’t paint learning
outcomes on
programs
Review programs, services, activities
Practice
SLOs
MetricsAssessment Plans
Organization
SourcesDomains
Sub-domains7
![Page 15: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/15.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
Linkages
Look upLook downLook aroundLook aroundLook down
Look upMissionContext
Colleagues
Priorities
Don’t paint learning
outcomes on
programs
Review programs, services, activities
Practice
SLOs
MetricsAssessment Plans
Organization
SourcesDomains
Sub-domains
Data
CollectionAnalysis
Reporting 7
![Page 16: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/16.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: content
PurposesConcepts
LearningOutcomes
MetricsAssessment
ReportingPlanning
Linkages
Look upLook downLook aroundLook aroundLook down
Look upMissionContext
Colleagues
Priorities
Don’t paint learning
outcomes on
programs
Review programs, services, activities
Practice
SLOs
MetricsAssessment Plans
Organization
SourcesDomains
Sub-domains
Data
CollectionAnalysis
Reporting
Utilization
ImprovementPlanning
Communications
Curriculum
7
![Page 17: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/17.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
Commitment
8
![Page 18: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/18.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
Structure
8
![Page 19: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/19.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
StructureLeadership linkages
8
![Page 20: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/20.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
StructureLeadership linkagesExpectations
8
![Page 21: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/21.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
StructureLeadership linkagesExpectationsProfessional
development
8
![Page 22: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/22.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
StructureLeadership linkagesExpectationsProfessional
developmentCollaborative feedback
8
![Page 23: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/23.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
StructureLeadership linkagesExpectationsProfessional
developmentCollaborative feedbackOffice hours
8
![Page 24: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/24.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
StructureLeadership linkagesExpectationsProfessional
developmentCollaborative feedbackOffice hours
Digital support
8
![Page 25: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/25.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
StructureLeadership linkagesExpectationsProfessional
developmentCollaborative feedbackOffice hours
Digital supportReporting retreats
8
![Page 26: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/26.jpg)
© 2007 Keeling & Associates, LLC
Capacity-building: process
CommitmentOrganizational
StructureLeadership linkagesExpectationsProfessional
developmentCollaborative feedbackOffice hours
Digital supportReporting retreatsReading &
resources8
![Page 27: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/27.jpg)
© 2007 Keeling & Associates, LLC
Culture change: language, mind-set, internal relationships, transparency, approach, style
9
Different words
Evidence-based
Systematic
Data
Feedback
Assesseverything
Out-comes
![Page 28: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/28.jpg)
© 2007 Keeling & Associates, LLC
Culture change: language, mind-set, internal relationships, transparency, approach, style
9
Different words
Evidence-based
Systematic
Data
Feedback
Assesseverything
Out-comes
Different words
Evidence-based
Systematic
Data
Feedback
Assesseverything
Out-comes
Systematic
Data
Feedback Culture of Evidence/Assessment with Commitment to
Improvement
![Page 29: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/29.jpg)
© 2007 Keeling & Associates, LLC
Building staff capacity at Georgia State
• New language and concepts
• Common format
• Consistent structure
• Skill-building with repetition
• Stretch and support
• Continuous feedback
• Cross-fertilization
10
![Page 30: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/30.jpg)
© 2007 Keeling & Associates, LLC
What do educators find difficult about writing student learning outcomes?
11
![Page 31: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/31.jpg)
© 2007 Keeling & Associates, LLC
What are learning outcomes?
12
• Learning outcomes define the goals of learning experiences (activities, courses, majors, curricula, engagement with post-secondary education)
• What any student should be able to do, know, or value as a result of engaging in that learning experience
• Not the same as satisfaction with education, professors, facilities, activities, or experiences; not the same as outputs, such as grades, GPA, retention rate, graduation rate
• Outcomes define impact -- how the student has changed.
• The achievement of learning outcomes (student success) measures institutional effectiveness.
![Page 32: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/32.jpg)
© 2007 Keeling & Associates, LLC
Learning outcomes have context
• Look up: institutional mission, vision, values
• Look down: where are you, why are you there, who are you serving, what are they here for
• Look around: see how your work “fits”
13
![Page 33: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/33.jpg)
© 2007 Keeling & Associates, LLC
Learning outcomes have multiple levels
• Institutional, or university: overall learning goals for the entire university experience [cognitive maturity, empowered citizenship, global competency...]
• Divisional: learning goals derived from the Student Affairs curriculum [integrated identity, personal maturity, mature relationships, humanitarianism...]
• Departmental: learning goals for specific groups of programs, services, or activities [knowledge of another culture’s language and customs...]
• Program: learning goal of a specific program or activity [ability to identify cultural etiquette in a particular situation]
14
![Page 34: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/34.jpg)
© 2007 Keeling & Associates, LLC
Institutional-level outcomes from Learning Reconsidered
• Cognitive complexity
• Knowledge acquisition, integration, and application
• Humanitarianism
• Civic engagement
• Inter- and intra-personal competence
• Practical competence
• Persistence and academic achievement
15
![Page 35: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/35.jpg)
© 2007 Keeling & Associates, LLC
Guidelines for departmental learning outcomes
• Very specific
• Practical
• Measurable
• Meaningful
• Consistent
• Sensible
• Credible
16
![Page 36: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/36.jpg)
© 2007 Keeling & Associates, LLC
What would make a student learning outcome impractical?
17
![Page 37: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/37.jpg)
© 2007 Keeling & Associates, LLC
Assessment planning
• Determine with as much precision and accuracy as possible exactly what is being assessed
• What group of students?
• What new capacity is sought?
• How will achievement of that capacity be demonstrated?
• What range of achievement satisfies the desired outcome?
• What external requirements, guidelines, or standards apply?
18
![Page 38: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/38.jpg)
© 2007 Keeling & Associates, LLC
Format for writing a student learning outcome
First year students
who will be able to
Graduating seniors
Students
participate in
engage with
complete
activity
program
course
8 counseling sessions
identify
list
describe
summarize
discuss
explain
as demonstrated by
interview
observed behavior
journaling
blog
portfolio
post-survey
at least 3 out of 4
70% accuracy
more/fewer than
all
[as required by]
external standards
intended elements
19
![Page 39: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/39.jpg)
© 2007 Keeling & Associates, LLC
Practice!
20
![Page 40: Putting Learning Reconsidered Into Practice: Developing ...assessment.uncg.edu/academic/docs/learning... · • Learning outcomes define the goals of learning experiences (activities,](https://reader031.fdocuments.us/reader031/viewer/2022011822/5ecd8e1e27e36d1e5c4968d9/html5/thumbnails/40.jpg)
© 2007 Keeling & Associates, LLC
Beyond the basics
• Embed theory in student learning outcomes
• Steering cognitive complexity to fit particular groups of students
• Curricular approach with student learning outcomes assigned differently for each year of matriculation
• Sophisticated approaches to reducing assessment burden
• Horizontally integrated learning outcomes help link programs, services, and activities
21