Models & Strategies for Assessing Institutional learning outcomes
description
Transcript of Models & Strategies for Assessing Institutional learning outcomes
![Page 1: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/1.jpg)
MODELS & STRATEGIES FOR ASSESSING INSTITUTIONAL
LEARNING OUTCOMES
Ashley Finley, Ph.DSenior Director of Assessment & Research, AAC&U
National Evaluator, Bringing Theory to Practice
Kutztown UniversityJanuary 19, 2012
LEARNING-CENTERED ASSESSMENT:
![Page 2: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/2.jpg)
WHY ARE WE HERE?
What are you doing?
Why are you doing it?
How do you know you’re doing it well?
What does it mean to be learning-centered at KU?
What practices promote and advance student learning?
How pervasive are these? What is the change you want
to see in your students when they leave the institution?
What are the foundational skills and competencies students should acquire throughout their learning experiences?
What are the effective approaches for implementing a cycle of inquiry and improvement?
![Page 3: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/3.jpg)
THE NATIONAL WORK ON LEARNING-CENTERED ASSESSMENT, LIBERAL EDUCATION OUTCOMES, & GENERAL EDUCATION
Liberal Education: A philosophy of learning that empowers & prepares individuals to deal with complexity, diversity, & change. = Broad knowledge combined w/ in-depth study To help students develop a sense of social responsibility,
strong & transferable intellectual and practical skills & a demonstrated ability to apply knowledge.
Vast majority of institutions have articulated learning outcomes
ANY institution can achieve the goals of a liberal education – 3 Major National Initiatives Compass Project (State University Systems – CA, OR, WI) Quality Collaboratives VALUE Rubric Project
![Page 4: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/4.jpg)
WHAT EMPLOYERS SAY ABOUT LIBERAL EDUCATION SKILLS:
88%
88%
90%
91%
% of Employers who agree with each statement
• Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past
• Employees are expected to work harder to coordinate with other departments than in the past
• The challenges employees face within our company are more complex today than they were in the past
• To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past
Source: Raising the Bar (AAC&U, 2010)
![Page 5: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/5.jpg)
70%
70%
71%
75%
75%
79%
81%
89%
% of Employers who agree that two- and four-year colleges should place MORE emphasis on helping students develop
the following:Effective oral/written
communicationCritical thinking/
analytical reasoningKnowledge/skills
applied to real world settings
Analyze/solve complex problems
Connect choices and actions to ethical
decisionsTeamwork skills/ ability
to collaborateAbility to innovate and
be creativeConcepts/developments
in science/technology
Source: “Raising the Bar: Employers’ Views on College Learning in the Wake of the Economic Downturn” (AAC&U and Hart Research Assoc. 2010)
![Page 6: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/6.jpg)
52%
57%
57%
63%
65%
67%
68%AND IN THESE AREAS…
Locate/organize/evaluate informationUnderstand global context of
situations/ decisionsGlobal issues’ implications for future
Understand & work with numbers/statistics
Understand role of U.S. in the worldKnowledge of cultural diversity in
US/worldCivic knowledge, community engagement
“Raising the Bar: Employers’ Views on College Learning in the Wake of the Economic Downturn” (AAC&U and Hart Research Assoc. 2010)
![Page 7: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/7.jpg)
0
30,000
60,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Mean Earnings: Use of Speaking Skills
Earni…
Source: Georgetown University, Center for Education and the Workforce (Anthony Carnavale)
HOW MUCH DOES A LIBERAL EDUCATION PAY?: “IMAGINATION IS MORE IMPORTANT THAN KNOWLEDGE”
-ALBERT EINSTEIN
![Page 8: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/8.jpg)
0
30,000
60,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Mean Earnings: Use of Originality
Earnings
Source: Georgetown University, Center for Education and the Workforce
![Page 9: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/9.jpg)
0
35,000
70,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Mean Earnings: Use of Judgement & Decision-Making Skills
Earnings
Source: Georgetown University, Center for Education and the Workforce
![Page 10: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/10.jpg)
0
30,000
60,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Mean Earnings: Use of Social Skills
Earnings
Source: Georgetown University, Center for Education and the Workforce
![Page 11: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/11.jpg)
Content Outcomes
Learning Outcomes
•Theoretical traditions•Methodology•Topical areas
How do we assess these?• Course Exams•Course Papers•Individual/ Group course projects•In-Class work/ participation
•????•????•????
Articulating Student Learning and Development
Kutztown Mission: “provide a high quality education at the undergraduate and graduate levels in order to prepare students to meet lifelong intellectual, ethical, social, and career challenges.”How do we assess these?
![Page 12: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/12.jpg)
• Knowledge of Human Cultures & the Physical &
Natural World• Content areas
• Intellectual & Practical Skills
• Inquiry & Analysis• Critical & Creative Thinking
• Written & Oral Comm.• Reading
• Quantitative Literacy• Information Literacy
•Teamwork & Problem-solving• Personal & Social Resp.
• Civic Knowledge• Intercultural Knowledge &
Competence• Ethical Reasoning• Lifelong Learning
• Integrative & Applied Learning
Gen.Ed./ Major
Institutional Outcomes
•Goal 1: Cultivate intell. & pract. skills• Inquiry & Analysis• Decision making• Written & Oral Communication• Quantitative Literacy• Information Literacy• Teamwork & problem solving• Wellness• Practiced throughout curr, w/ progressively
more challenging probs, projects & standards
•Goal 2: Develop understanding of human cultures & physical and natural world• Sciences, mathematics, social sciences,
humanities, histories, languages, arts • Engage w/ big questions, contemp. &
enduring•Goal 3: Sense of personal & social resp.• Local/Global civic knowledge • Intercultural knowledge & competence• Ethical reasoning & action• Personal qualities & attitudes (passion,
curiosity, self-confidence, imagination, cooperation, commitment, support)
• Active inv. w/diverse comm. & real world chall.
![Page 13: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/13.jpg)
Learning Outcome
s
Assess Learning Outcome
s
What is the story you want to tell about student learning?
![Page 14: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/14.jpg)
Outcomes
AssessmentCurriculum
![Page 15: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/15.jpg)
Learning Outcome
s
Assess Learning Outcome
s
What is the story you want to tell about student learning?
![Page 16: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/16.jpg)
Basic Logic ModelINPUTS
Resources are needed
for activities –
What is needed to
start or keep
something going?• Human• Financial•Technological• Curricular/
Program
ACTIVITIES
Actions or processes
necessary to produce outputs –
What activities need to occur to produce
evidence?
OUTPUTS
Products used to assess
outcomes - What can
be counted as evidence of change?
OUTCOMES
Expected Changes and
Benefits –What are the
goals you want to reach?
• Short-term
• Intermediate• Long-term
Assumptions
![Page 17: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/17.jpg)
OUTCOMES• Goals &
Domains of Learning
• Understanding of human cultures…
• Intellectual and practice skills• Critical
thinking• Personal &
Social Responsibility• Civic
engagement
Sample Logic Model INPUTS•GE•Depts.•Advising•Writing Center•Stud. Aff.•Health & Wellness Center•Alumni Engagement•Multicult. Services•Athletics•Distance Ed•Career Services•Inst. Res.
ACTIVITIES•Experiential learning•Civic learning•Interaction
OUTPUTS•Signature assignments•Reflection papers•Group projects•Community-based projects
•Engagement in life’s big questions•Practiced extensively & progressively•Active involve. In diverse comm. & real-world challenges
![Page 18: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/18.jpg)
VALUE PROJECT (WWW.AACU.ORG/VALUE) Project Goals
1) Create dialogue and develop shared understanding of common learning outcomes
2) Create template for direct assessment of student work (in text and non-text formats)
3) Create student-friendly format to engage students in self-evaluation
Rubric Development Teams of faculty/scholars nationwide
Reviewed existing rubrics to identify commonalities, clarify language, and develop broad agreement on outcomes criteria (openedpractices.org)
To date accessed by over 3000 institutions/organizations, 11,000 individuals Domestic & international, K-12, state systems
Reliability study
![Page 19: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/19.jpg)
LIST OF VALUE RUBRICS Knowledge of Human
Cultures & the Physical & Natural Worlds Content Areas No Rubrics
Intellectual and Practical Skills Inquiry & Analysis Critical Thinking Creative Thinking Written Communication Oral Communication Reading Quantitative Literacy Information Literacy Teamwork Problem-solving
Personal & Social Responsibility Civic Knowledge &
Engagement Intercultural
Knowledge & Competence
Ethical Reasoning Foundations & Skills
for Lifelong Learning Integrative &
Applied Learning Integrative &
Applied Learning
![Page 20: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/20.jpg)
Located at: http://www.aacu.org/value/index.cf
m
![Page 21: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/21.jpg)
![Page 22: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/22.jpg)
Criteria
The Anatomy of a VALUE Rubric
Levels
Performance Descriptors
![Page 23: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/23.jpg)
THE CALIBRATION TRAINING PROCESS Scoring Steps:
Review rubric to familiarize yourself with structure, language, performance levels
Ask questions about the rubric for clarification or to get input from others regarding interpretation
Read student work sample Connect specific points of evidence in work sample with
each criterion at the appropriate performance level (if applicable)
Calibration Steps: Review scores Determine common score(s) Hear from outliers Discuss Determine final score
![Page 24: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/24.jpg)
THE GROUND RULES This is not grading. We are not changing the rubric (today). Our work is time sensitive. Go with your
instinct. Think globally about student work and about
the learning skill. Think beyond specific disciplinary lenses or content.
Start with 4 and work backwards. Pick one performance benchmark per criterion. Avoid “.5”.
Zero does exist. Assign “0” if work does not meet benchmark (cell one) performance level. N/A exists. Assign “not applicable” if the student work is not intended to meet a particular criterion.
![Page 25: Models & Strategies for Assessing Institutional learning outcomes](https://reader036.fdocuments.us/reader036/viewer/2022062501/56815d11550346895dcb1238/html5/thumbnails/25.jpg)
SIGNATURE ASSIGNMENTS Assignment should enable attainment of
criteria Break down criteria to determine key components for
assignment What should students do with content to
meet criteria? E.g. What are the pieces to be analyzed, compared,
integrated? Will the assignment be used for more than
one outcome? What are the types of assignments that will
be most helpful for allowing students to demonstrate competency?