Assessment of Learning Objectives in Study Abroad

An Integrative Model

Diane Jonte-Pace, Santa Clara UniversityBarbara Molony, Santa Clara University

Joan Gillespie, IES

IES April 8, 2010

Assessment and accreditation

New expectations for accountability/ transparency /assessment– National regulations– Regional accrediting bodies

Change is hard– Challenges on university campuses– Further challenges face Study Abroad

SCU offers meaningful & manageable assessment model– Designed during re-accreditation process– Continues in systematic multi-year cycle

A proposal

Step 1: Fall 2010 Pilot– Integrate IES Rome into SCU Core assessment plan

– One Religious Studies course

Step 2: Future years– Integrate SCU model into IES practices more broadly

Accreditation and Assessment

Accreditation Agencies request assessment of student learning

Middle States (MSCHE) New England (NEASC-CIHE) North Central (NCA-HLS) Northwest commission (NWCCU) Southern Association (SACS) Western Association (WASC

Examples The institution’s student learning outcomes and expectations for student

attainment are clearly stated at the course, program and, as appropriate, institutional level. WASC Standard 2.3

The institution’s faculty takes collective responsibility for establishing,

reviewing, fostering, and demonstrating the attainment of expectations for student learning and attainment. WASC Standard 2.4

Key Concepts for SCU

Goals and Objectives– Broad Goals– Goals– Measurable Learning Objectives

Course level & program level

Explicit assignment mapping

Direct vs. Indirect assessment

Program Level Assessment

Rubric Scoring vs Grading

Systematic multi-year assessment plan

Similar to IES Key Concepts

Learning Outcomes – Each syllabus should list the expected outcomes for the

course, i.e. what the student is expected to know and be able to do upon completion of the course.

Required Work & Assessment– The required work and content should be clearly linked

to learning outcomes.

Assessment– Grading, percentages

Other IES Concepts and Practices

Knowledge

Global Cultures

Arts & Humanities

Scientific Inquiry

Science & Technology

Diversity

Civic life

Habits of Mind and Heart

Critical Thinking

Mathematical Reasoning

Complexity

Ethical Reasoning

Religious Reflection

Communication

Engagement with the World

Perspective

Collaboration

Social Justice

Civic Engagement

Broad Learning Goals SCU

SCU goals distributed across multiple Core areas

Selected Core Areas

Goals

Religion, Theology, Culture

Civic Engagement

Cultures & Ideas 3

Knowledge Civic Life Global Cultures

Diversity

Habits of Mind & Heart

Crit Thinking

Complexity

Religious Reflection

Communicat. Critical Thinking

Engagement with the World

Collaboration Civic Engagement

Perspective

From Goals to Objectives at SCU

Faculty Core Committees translate goals into measurable program learning objectives for Core Area

Bloom’s taxonomy of cognitive development provides framework– Identify, Describe, Recognize– Compare, Apply, Analyze– Evaluate, Synthesize

All syllabi in Core Area include Area (Program) Objectives

Goals transformed into objectivesReligion, Theology & Culture 2

Goals: Critical thinking, complexity, religious reflection

Objectives: Students will:

2.1 Analyze complex and diverse religious phenomena, such as architecture and art, music, ritual, scriptures, theological systems, and other cultural expressions of religious belief.

2.2 Integrate and compare several different disciplinary approaches to a coherent set of religious phenomena.

2.3 Clarify and express beliefs in light of their critical inquiry into the religious dimensions of human existence.

Assignment Mapping: An Example from SCU

The first paper will ask students to focus on a single issue in Hinduism (e.g. spiritual practice, religious iconography, nature of self & divine) from a single disciplinary perspective (Core LO 1)

The second paper will ask students to analyze a single religious phenomenon such as a devotional song or icon from multiple disciplinary perspectives (e.g., theological, historical, ethnographic, art historical) (Core LO 1 & 2)

The third paper will focus on the experience of contemporary Hindus, with a focus on belief and practice in communities centering around female gurus in India and abroad (Core LO 1 & 3)

SCU multi-year assessment plan (partial list)

Fall 2010 Religion Theology Culture 2

Fall 2011 Natural Science

Fall 2012 Civic Engagement

Fall 2013 Experiential Learning Soc Justice

Fall 2014 Cultures & Ideas 3

Fall 2015 Pathways

Proposal, Step 1Include IES Rome Religion Course in Multi-year Assessment Plan

IES RL 435 Monotheisms: The Children of Abraham – Term paper– Midterm– Final

Fall 2010, ~20 SCU students enrolled– All student work for targeted learning objective

included in SCU assessment

How would the pilot assessment project work?

One learning objective selected for assessment

At SCU, work gathered from randomly selected students in each RTC2 class for selected learning objective

In Rome (with fewer students), all student work gathered for selected learning objective

Rubric scoring party January 2011

Invite IES to SCU rubric scoring party via tele-conference

Close feedback loop: Reflect on assessment reports

Goal: Enhance teaching & learning

Rubric Scoring Parties

6-10 volunteers

2-3 hours

Rubrics for one (or two) learning objectives

Readers calibrated with common readings

Two readers per paper; third reader if needed

Food

Proposal Step 2 Integrate SCU model into IES more broadly

Learning objectives at the program level

Explicit assignment mapping at the course level

Direct assessment of selected student workRubric Scoring

Systematic multi-year assessment plan

Challenges

Aren’t SCU’s processes and assumptions different from IES’s?

Logistical obstacles: distance, language, distinct pedagogical culture in each IES site

Challenges not insurmountable IES shares practices with SCU

SCU and IES Learning Goals are aligned with national vision of best practices

Logistics: Exciting opportunity for global collaboration through systematic plan for rubric development

SCU & IES: Shared Vision & Practice

IES 3-D paradigm similar to SCU’s Goals– Cognitive Goals/Ends– Interpersonal Goals/Ends– Intrapersonal Goals/Ends

IES Syllabus Guidelines already require assignment alignment

IES is committed to ongoing assessment & improvement of student learning

IES 3-D Program Model & SCU’s Core

Means

Ends

Curriculum Cocurriculum Community

Cognitive Knowledge

Interpersonal Habits of Mind & Heart

Intrapersonal Engagement with the World

Knowledge/ Cognitive Ends

Global Cultures

Arts & Humanities

Scientific Inquiry

Science & Technology

Diversity

Civic life

Habits of Mind and Heart/ Interpersonal Ends

Critical Thinking

Mathematical Reasoning

Complexity

Ethical Reasoning

Religious Reflection

CommunicationEngagement with the World/

Intrapersonal Ends Perspective

Collaboration

Social Justice

Civic Engagement

SCU Learning Goals with IES “Ends”

Alignment with National Vision

AAC&U IES SCUKnowledge of Human Cultures

Intellectual and Practical SkillsInquiry and analysisCritical and creative thinkingWritten and Oral CommunicationInformation LiteracyTeamworkProblem Solving

Personal and Social Responsibility

Integrative and Applied Learning

Cognitive

Interpersonal

Intrapersonal

Knowledge

Habits of Mind & Heart

Engagement with the World

IES & SCU: Similar PracticesIES

Require clear learning objectives on syllabi at course level

Request alignment of assignments

Require assessment of student learning at course level

SCU

Require learning objectives on syllabi at course & program level

Require explicit assignment mapping

Assess student learning at program level as well as course level

Major Difference: Program Level Assessment

How does Program Level Assessment work?

Determine Program Level Objectives

Select program level learning objective for assessment

Develop rubric for assessment of study work

Gather randomly selected student work

Host 2-3 hour rubric scoring party, assess student work for selected learning objective

Provide feedback to participating departments

IES incorporates SCU practices

1. Develop program level learning objectives

2. Include program level learning objectives on syllabi

3. Map assignments explicitly to learning objectives

4. Develop multi-year assessment plan

Logistics of Proposal

Summary

Fall 2010 SCU includes IES Rome course in Core Assessment plan for Religion, Theology, Culture 2

IES incorporates key aspects of SCU assessment model– Measurable Learning Objectives at program level– Explicit assignment mapping– Multi-year assessment plan Direct assessment of student work in rubric scoring parties

Feedback; reflection; improvement of teaching & learning

SCU includes SCU student work from other IES sites in multi year assessment plan

Contact us Office of Undergraduate Studies

Santa Clara University

www.scu.edu/assessment

www.scu.edu/core

Diane Jonte-Pace, Vice Provost, Undergrad Studies

Carol Ann Gittens, Director of Assessment

Barbara Molony, Chair, History Dept; SCU Core Coordinator; IES Curriculum Committee member

Joan Gillespie, Associate Vice President of Academic Affairs, IES