2010 Teaching and Learning Symposium Taking the Next Step in Outcomes Assessment: Surveying Alumni...

2010 Teaching and Learning Symposium

Taking the Next Step in Outcomes Assessment: Surveying Alumni to Assess the Long-Term Impact of Study Abroad

Rob Howell, Sally Magnan, Dianna Murphy, Alice Astarita, Carolina Bailey

Impact of Study Abroad: IAP Participants

2005-2006 2009-2010

IAP Participants 909 1287 (est.)

Impact of Study Abroad: IAP Programs

2005-2006 2009-2010

IAP Programs 86 161

Impact of Study Abroad: Program Duration

Program Duration

2005-2006 2009-2010

Year 133 131

Semester 489 747

Short-term 287 409

Impact of Study Abroad: World Regions

2005-2006 2009-2010

Europe 47 74

Asia 18 33

Latin America 11 23

Africa 6 11

Middle East 1 7

Oceania 2 9

North America 0 2

Current IAP Assessments

• Doctoral dissertation research on acquisition of language and cultural competence

Need for Project

• What are the long-term educational outcomes of study abroad with respect to ongoing global engagement throughout life?

• How does the level of global engagement of former study abroad students differ from that of alumni who did not study abroad?

SAGE: Study Abroad for Global Engagement

• Long-term impact of study abroad– Beyond personal and professional impact: Social benefit, common

good– Civic and public engagement– Behavioral manifestations

• Retrospective tracer study: 1960-2005• Questionnaire, follow-up interviews • 6,391 alumni, 22 U.S. colleges, universities, study abroad• R. Michael Paige, Gerald W. Fry, University of Minnesota• U.S. Department of Education grant

(2006-2009)

Paige et al. (2009): Conceptual Model for Global Engagement

Five dimensions of global engagement:• Civic engagement• Voluntary simplicity• Knowledge production• Philanthropy• Social entrepreneurship

SAGE: Select Major Findings

• Study abroad a major impact on participants’ lives: one of the most important of the undergraduate experience

• Alumni of study abroad programs self-report behaviors of global engagement in ways “unlikely to pertain to larger population” (Paige et al. 2009, 41)

• Study abroad perceived by respondents as influential on later global engagement activities

Drawback to initial SAGE study: No control group

UW-Madison Study: Life Experiences of UW-Madison Alumni

Goal: Assess long-term impact of UW-Madison study abroad programs on global engagement of alumni by comparing global engagement of alumni who did and didn’t study abroad as undergraduates

People, Funding

• Collaborative assessment project – Language Institute: Sally Magnan, Dianna Murphy– International Academic Programs (IAP); Rob Howell – Doctoral Program in Second Language Acquisition (SLA):

Alice Astarita, Carolina Bailey

• 2010-11 funding: Office of the Provost, University Assessment Funds; IAP

Research Questions

1. Is there a difference in the overall global engagement of alumni who did and didn’t study abroad as undergraduates?

2. Is there a difference, by dimension (category) of global engagement, of alumni who did and didn’t study abroad?

Study Design

• Retrospective study of UW-Madison alumni (10 years)

• Written (online) questionnaire• Targeted study abroad programs for 2010-11

research:– Florence, Italy– Bologna, Italy– Aix-en-Provence, France– Freiburg, Germany– Madrid, Spain

Study Design

• Retrospective study of UW-Madison alumni• Written (online) questionnaire• Targeted programs for 2010-11 research:

– Florence, Italy– Bologna, Italy– Aix-en-Provence, France– Freiburg, Germany– Madrid, Spain

Study Abroad Program in Florence

• Duration: One semester• Program type: Island• Location: Sesto Fiorentino, 6 miles from Florence,

Italy; renovated 16th century Villa Corsi-Salviati• Curriculum:

• Courses in English by U.S. faculty• Internships• Group trips

Questionnaire

• Written (online) questionnaire, based on SAGE instrument

• 67 items; fixed and free response• Additional dimension of global engagement:

Internationally-oriented personal enjoyment activities:– Watch films in languages other than English– Host international visitors– Read international newspapers– Travel internationally for pleasure

Participants

• Participants– UW-Madison alumni of Florence program (1996-

2009); graduated 2001-2009 (n=46)– Random sample of UW-Madison alumni (graduated

2001-2009), did not study abroad; similar characteristics to target sample in terms of year of graduation, major school/college (n=153)

Data Collection Procedures

• Procedures– Alumni lists (IAP)– Contact information, year of graduation, major

school/college (Wisconsin Alumni Association, WAA)– Email invitations, links to online questionnaire (WAA)

• 363 alumni of Florence program• 2,188 alumni who did not study abroad

– Response rate 14% (studied abroad); 9% (did not study abroad)

Exempt from IRB Review (SE-2009-0790)

Participants: Year of Graduation

Participants: Major School/College

Studied abroad(n=46)

Did not study abroad (n=153)

Letters and Science 71% 63%

Journalism and Mass Communication 8% 7%

Human Ecology 6% 7%

Education 4% 9%

Agricultural and Life Sciences 4% 5%

Social Work 4% 2%

Other 2% 9%

Influence of Study Abroad on Global Engagement

College experience Large/some degree

Athletics/intramural sports 12%Community service/volunteer work 57%

Coursework 76%

Fraternity/sorority 10%Friendships/student-peer interactions 76%

Interaction with faculty 60%

Internships in the United States 26%

Religious organizations 12%

Student clubs 46%

Student government 12%

Study abroad 81%

Work/employment during college 45%

Data Analysis Procedures

1. Descriptive data: Percentages of positive responses 2. Comparison of responses of alumni who did and did not study

abroad:• Data values in different scales

– 4-point (Never/Rarely/Sometimes/Frequently; Not at all/Very Little/Some Degree/Large Degree)

– 2-point (Yes/No)– Average (Free response: Hours/week volunteer; $/year donations)

• Scales normalized: responses converted to z-values• T-tests for difference between groups run on

z-values • Alpha level for significant difference .05

Research Question 1: Difference in Overall Global Engagement?


Sig. (2-tailed T-test) .002

N Mean of z-value

Studied abroad 46 .1627

Did not study abroad 153 -.0489

Research Question 2: Difference in Global Engagement, by Dimension of Global Engagement?

2. Is there a difference in the different dimensions of global engagement of alumni who did and didn’t study abroad as undergraduates, by category?

Type of global engagement Sig. (2-tailed)

Civic engagement .014

Voluntary simplicity .015

Knowledge production Difference not significant

Philanthropy Difference not significant

Social entrepreneurship Difference not significant

Internationally-oriented personal enjoyment activities

.015

Civic Engagement: On Issues of Domestic Importance...

N=46

Civic Engagement: Degree to Which Study Abroad Was Helpful in Developing Domestic Civic Engagement

Civic Engagement: On Issues of International Importance...

Civic Engagement: Degree to Which Study Abroad Was Helpful in Developing International Civic Engagement

Voluntary Simplicity: Practice Voluntary Simplicity

Studied abroad: n=46Didn’t study abroad: n=153

Voluntary Simplicity: Degree to Which Study Abroad Influenced Practice of Voluntary Simplicity

Internationally-Oriented Personal Enjoyment Activities

Studied abroad: n=46Didn’t study abroad: n=153

Degree to Which Study Abroad Was Helpful in Developing Internationally-Oriented Personal Enjoyment Activities

Not at all 2%

Friendships With People of Cultures Different than Own

Impact of Study Abroad: Free Response

• Study abroad significantly changed my life--it opened the whole world up to me, and not in the cliché and obvious way, but in exciting and shocking ways... I was challenged to become kinder, a better listener, more skilled in asking for what I needed and conveying my concerns and fears... It taught me to be responsible for myself, my country, my world, and that each action I took or that my country took had repercussions much larger than I could have ever imagined. It absolutely changed my life and led me to the field I am in today-public diplomacy.

Impact of Study Abroad: Free Response

• Study abroad was literally the best thing I've ever done, undoubtedly the best decision I've ever made, and ignited a passion for travel and other cultures in me like few other passions in my life. It's been a conversation starter and stronghold, a foot in the door and resume booster, and it was the beginning of friendships I still cherish today.

Impact of study abroad: Free response

• It was a great experience. It opened the doors for me for future travel. After I graduated from UW-Madison I spent a year travelling and working abroad. I wouldn't have done that had I not had my study abroad experience. It also increased my interest in international issues and careers.

• I wouldn't trade that experience for anything... It wasn't all fun -- getting used to cultural differences took time -- but in the end made me a more well-rounded, tolerant person (not to say I wasn't before I left). Studying abroad gave me a world view that you can't learn in a classroom in the States.

Unsolicited response from participant

• I just wanted to let you know that I thought that survey was well-written and I really enjoyed filling it out. Man, I love being an alumni of this school!

Limitations

• Limited type(s) of programs• Near- vs. long-term impact

– Knowledge production– Philanthropy– Social entrepreneurship

• Pre-disposition• Self-reporting, self-awareness/memory

Implications

• Social benefits of study abroad – Institutional support – Access/participation– Undergraduate curriculum

• Program design

Next Steps

• Programs outside of Europe– College Year in India (Varanasi, India)– College Year in Thailand (Chiang Mai, Thailand)– Other

• Program design– Island– Sheltered– Direct-enroll

• Sample– Long-term– Near-term

• Variables other than study abroad participation– Socioeconomic status– Prior international experience(s)– Expressed interest in study abroad

Prior international experience

Major(s) school or college

Year of graduation:1-5 years ago

6-10 years ago

Internationally-oriented career

Advanced degree

Civic engagemen

t

Knowledge production

VoluntarySimplicity

Philanthropy

Social entrepreneur

ship

Income

Background Demographic

Variables

Explanatory Variables

Outcome: Global Engagement

Outcome: Advanced

Degree / Career

Internationally-oriented personal

enjoyment activities

Studied abroad

Expressed interest in study abroad,

but did not go

Did not study abroad

Next steps: Analysis

Socio-economic status/ Parental income / Parental education

2010 Teaching and Learning Symposium

Taking the Next Step in Outcomes Assessment: Surveying Alumni to Assess the Long-Term Impact of Study Abroad

Rob Howell, Sally Magnan, Dianna Murphy, Alice Astarita, Carolina Bailey

Research Question 1: Difference in Global Engagement?


Studied Abroad Did Not Study Abroad

98% 26%

Research Question 2: Difference in Global Engagement, by Type?

2. Is there a difference in the different categories of global engagement of alumni who did and didn’t study abroad as undergraduates, by dimension of global engagement?

Type of global engagement

Studied Abroad Did Not Studied Abroad

Civic engagement 61% 46%

Voluntary simplicity

62% 46%

Knowledge production 51% 49%

Philanthropy 55% 48%

Social entrepreneurship

50% 50%

Internationally-oriented personal enjoyment activities

58% 47%

2010 Teaching and Learning Symposium Taking the Next Step in Outcomes Assessment: Surveying Alumni...

Documents

Transcript of 2010 Teaching and Learning Symposium Taking the Next Step in Outcomes Assessment: Surveying Alumni...