ELPS 431, Evaluation in Higher Education
-
Upload
kstephens73 -
Category
Documents
-
view
222 -
download
0
Transcript of ELPS 431, Evaluation in Higher Education
-
8/12/2019 ELPS 431, Evaluation in Higher Education
1/91
1Running head: EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Evaluation Proposal:
The Chicago Center Program
Abbie A. Ray and Katie C. Stephens
Loyola University Chicago
December 2, 2013
-
8/12/2019 ELPS 431, Evaluation in Higher Education
2/91
2EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Table of Contents
Setting and Context of the Chicago Center Program ...3 -4
History of the Chicago Center Program ...4 -5
Problem the Chicago Center Program was designed to correct ...5 -6
Goals of the Chicago Center Program ..6 -7
Program Components and Organization ...7 -8
Chicago Center Program Stakeholders . ..8-11
Contextual Circumstances ....11 -12
Overriding Questions .... .12
Logic Model .12 -17Evaluation Approach .... 17-20
Quantitative Methods ...20 -25
Qualitative Methods .....25 -31
Limitations ....31 -32
Next Steps .....32 -33
References34 -35
Appendices:
Appendix A: Previous Evaluations ...36-43Appendix B: Exploring the United States through Chicago , Course Documents ....44-50Appendix C: Logic Model ....5 1-52Appendix D: Pre-Test ...5 3-55Appendix E: Post-Test ..5 6-61Appendix F: Notification e-mail of in-class survey (post-test) .6 2Appendix G: Survey Construct Map Pre-Test ...6 3-64Appendix H: Survey Construct Map Post-Test .....6 5-69Appendix I: Focus Group Invitation E-mail .. 70Appendix J: Focus Group Protocol .. 71-74Appendix K: Focus Group Consent Form .....7 5Appendix L: Note-Taking Sheets .....7 6-82Appendix M: Coding Rubric 8 3-84Appendix N: Evaluation Budget ...8 5Appendix O: Evaluation Timeline .8 6Appendix P: PowerPoint Presentation ..87-91
-
8/12/2019 ELPS 431, Evaluation in Higher Education
3/91
3EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
The Chicago Cent er Program at Loyola University Chicago serves as the campus study
abroad opportunity for a group of students seeking an education abroad experience similar to
what an American student experiences when studying abroad. Chicago Center Program staff
plans programs for this specific group of students and provides them general support throughout
their time at Loyola University Chicago. As a new program to campus, the Chicago Center
Program is experiencing a time of growth as it gains attention both on community and global
levels. The evaluation plan being presented will provide context for the program along with an
in-depth history for the reader. Following the program context and historical pieces will be a
detailed evaluation plan that includes the quantitative and qualitative methods. Finally,
appendices will serve as supplements to the narrative. At this time, the evaluation is intended to
provide stakeholders with insight to the programs current practices as well as explore areas for
improvement.
Setting and Context of the Chicago Center Program
The Chicago Center Program office at Loyola is located in the International House on the
Loyola Lake Shore Campus in the historic Chicago neighborhood of Rogers Park. The Chicago
Center Program itself is conducted through several mediums such as in-class work, excursions
across the United States, and in-house programming. The International House houses all
program participants during their stay in the United States. The program also reaches several
off-site locations such as Washington D.C, Los Angeles, New York, and New Orleans. In these
cities, students have the opportunity to be immersed in different regions of the U.S. to experience
other U.S. cultures for five-day period. The program also takes place on the campus itself inside
the classroom. In addition to their academic schedules, many students participate in co-curricular
activities during their time in the United States. Currently, the International Students that
-
8/12/2019 ELPS 431, Evaluation in Higher Education
4/91
4EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
participate in the Chicago Center program originate from three countries: Brazil, Korea, and
China. The total enrollment for the fall semester of 2013 consists of 10 Brazilians, 8 Koreans,
and 12 Chinese students. In regards to context, it is important to note that the Chicago Center
Program exists at a private, Jesuit institution in a large metropolitan city. Loyola University
Chicago has approximately 10,000 undergraduate students and the Chicago Center Program
encompasses 30 of those students (Garanzini, 2013). To speak to generally, a promise of Loyola
Universitys institution is that every student will be exposed to global awareness and be able to
demonstrate an understanding that the worlds people and societies are interrelated and
interdependent (Mission & Identity, 2013). Program s such as the Chicago Center intend to
demonstrate the promises of this mission by welcoming International Students to campus and
engaging them in cultural exchange between other non-U.S. students as well as interacting with
domestic students at Loyola.
The Chicago Center Program is to be implemented every fall and spring semester; it does
not offer a summer session. The Vice Provost and Director of the Chicago Center Program work
year-round, but the program does not currently have the budget to have a Graduate Assistant
during the summer. Participants of the Chicago Center Program have the choice to study at
Loyola University for either one semester or a complete a full year.
History of the Chicago Center Program
The Chicago Center Program started in Fall 2011 with five Chinese students from Beijing
University of International Business and Economics (UIBE) who expressed interest in having an
abroad experience in the Chicago similar to what students from Loyola were experiencing at
UIBE, the location of the Beijing Center. The Vice Provost recognized study abroad in the
United States for International Students as a deficit in the field of Education Abroad. He saw it as
-
8/12/2019 ELPS 431, Evaluation in Higher Education
5/91
5EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
an interesting area to explore with this market of students and their desire to study in the U.S.
without the support of a third-party provider. The Vice Provost created a tuition-based program
that would allow these five students to have a study abroad experience at Loyola University
Chicago that would be unique to the current offering of semester exchange programs from other
U.S. institutions. The pilot program was successful and has since expanded to other Chinese
universities, as well as Brazil, Korea and Pakistan. The Chicago Center Program currently hosts
30 students, but staff members are actively looking to expand the program in hopes it will
continue to grow in the coming years (personal communication, September 4, 2013).
Previous Evaluations
The Chicago Center Program conducted informal evaluations in the past containing
questions that were superficial and surface level. The previous evaluations focused on the
excursion trips and general level of happiness with the Chicago Center Program, but never
discussed topics in relation to learning outcomes or inter-cultural competence (please refer to
Appendix A).
Problem the Chicago Center Program was designed to correct
The Chicago Center Program is designed to provide an opportunity for students looking to
study in the United States for a shorter period of time than a four-year-degree with the benefit of
globalizing the campus at Loyola University Chicago and offering an all-encompassing program
similar to their American counterparts. The lack of non-degree programs in the United States for
students enrolled in degree-seeking programs at their home universities is an open market that
the Chicago Center Program seeks to fulfill. The majority of study opportunities available for
International Students in the United States are tailored toward students that aim to complete a
full degree at a U.S. institution. Open Doors Data illustrates the disproportionate statistics of
-
8/12/2019 ELPS 431, Evaluation in Higher Education
6/91
6EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
degree and non-degree seeking students studying in the United States. For example, during the
2012-2013 academic year, 651,197 International Students, both undergraduate and graduate,
studied in the U.S. as degree seeking students whereas only 73,528 identified as non-degree
seeking (Open Doors Data, 2013).
A large portion of American universities will allow students to study on campus for a
short period such as a semester or a year-long experience, but they do not necessarily have the
same resources available to them when compared to the study abroad experience for the typical
American student abroad. When many American students go abroad for an immersion
experience, there are typically staff members that work directly with them, specific courses that
they are required to take with other International Students and either live with host families or
have a special living arrangement (Watson, Siska, & Wolfel, 2013). A common experience of an
international student coming to the U.S. for a semester of study is to take classes at a university
and to be treated as if they were regular, domestic students at the university even though they
have completely distinct identities.
Goals of the Chicago Center Program
The Chicago Center Program at Loyola University Chicago serves to accomplish goals
related to opportunities in study abroad and cultural exchange. Loyola University Chicago
currently serves 10,168 undergraduate students and of that population, 636 are international
students (personal communication, November 22, 2013). The Chicago Center Program is an
opportunity provided by Loyola University Chicago to international students who are looking for
a study abroad experience in the United States. The Chicago Center Program is modeled off the
successful Loyolas Beijing Center in China in that it provides a similar study abroad e xperience
to international students through the support of knowledgeable staff that organizes programs on
-
8/12/2019 ELPS 431, Evaluation in Higher Education
7/91
7EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
campus around the topics of globalization and cultural exchange as well as exploring other
aspects of U.S. culture through excursions to other cities. Due to a growing demand in American
higher education to globalize the university experience, programs like the Chicago Center also
serve as a way for domestic students to have direct experience with another culture(s) without
having to leave campus (Salisbury, Umbach, Paulsen & Pascarella, 2009). In addition to the
benefits of cultural immersion and improvement in English language skills that International
Students experience while on a U.S. campus, domestic students can also utilize this relationship
as a learning opportunity in cultural exchange. The presence of International Students on
campus can provide opportunities for cross-cultural interaction that are reciprocal and mutually
beneficial (Twombly, Salisbury, Tumanut and Klute, 2012).
Program Components and Organization
The Program
The Chicago Center Program offers students from outside the United States an
opportunity to study at Loyola University Chicago for one semester or a year-long experience
alongside Loyola students and other students from around the world. Students may take up to
four courses on campus in addition to one required course which is a seminar designed
specifically for the Chicago Center Program titled: Exploring the United States through
Chicago. This seminar, only available to Chicago Center Program students, provides them with
the opportunity to discuss contemporary issues in the U.S. such as politics, race, and culture
while using Chicago as the lens. Chicago Center program participants who study at Loyola for
the full year are enrolled in the seminar for their first semester, but then continue their second
semester without the seminar course and take four courses. In order to provide a diversified
immersion of U.S. culture, students participate in two excursions per semester to other large
-
8/12/2019 ELPS 431, Evaluation in Higher Education
8/91
8EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
cities including Washington D.C. and New York City. Activities on these trips vary from visits
to historical sites to social outings that serve as an outlet for the students to get to know one
another better. The course syllabus provided to the participants during the first class session
provides an overview of the events for the semester (see Appendix B for syllabus and additional
course documents).
The on-campus experience of Chicago Center students is enriched through connections
with Host Students who are Loyola students that vol unteer to serve as mentors and cultural
liaisons for the International Students. The Host Students, most of who have studied abroad,
may be the first American relationships that the Chicago Center Program students form. They are
available for logistical purposes such as answering day-to-day questions like where to purchase
textbooks or how to use public transportation. They are also available for informal socializing
outside of the Chicago Center Program. A new feature of the 2013-2014 academic year is the
International House. All of the Chicago Center Program students reside together in the
International residence hall with other International Students at Loyola, who are not a part of the
Chicago Center Program, to create an international community on campus. Host Students are
also invited to activities and programs organized by Chicago Center Staff in order to provide
more domestic students with direct and reciprocal cultural experience.
Chicago Center Program Stakeholders
Chicago Program Staff
The main staff of the Chicago Center Program include: Dr. Patrick Boyle, the Vice
Provost for Academic Centers and Global Initiatives; Jason Obin, the Director of Chicago Center
Program; Katie Stephens, the Graduate Assistant; and Michael Hines, the Professor of the
seminar class all students are required to take. The Vice Provost is the leader of the program and
-
8/12/2019 ELPS 431, Evaluation in Higher Education
9/91
9EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
all other Chicago Center Program staff report to him. The Vice Provost is responsible for
ensuring that the program meets the expectations and standards of Loyola and of the partner
organizations that send students to the Chicago Center Program. He also is an integral
component in making connections with partner organizations from around the world. He gives a
voice to the strategic goals of the program to the university.
The Director of the Chicago Center Program manages the direction and the overall quality
of the program to ensure that students have a successful semester in regards to language growth,
global competence and intercultural understanding. The Director works with the professor of the
seminar to develop the course curriculum to ensure that it covers the necessary topics. The
Directors main responsibilities include making connections with potential partner schools in
other nations for recruiting students. This is done through marketing and publicity as well as
traveling to other countries to meet potential partners and give presentations to school
administrators and students. The Director also co-leads excursions to other cities and plays a
role in planning the trip itineraries and making other arrangements alongside with the Graduate
Assistant.
The Graduate Assistants role in the program is to plan the exc ursions to other cities and
co-lead these trips with the Director. Other job duties include coordinating the Host Student
program, working on expanding the programs aud ience via social media and serving as a
Teaching Assistant to the professor of the required seminar class. The p rofessors duties are to
teach the seminar class where students discuss current issues in the United States while exploring
Chicago as well as organize weekend field trips for the students in Chicago.
Direct Stakeholders
-
8/12/2019 ELPS 431, Evaluation in Higher Education
10/91
10EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
There are several key direct stakeholders involved in the program who benefit based on
the improvement and/or success of the program. To begin, the Vice Provost for Academic
Centers and Global Initiatives at Loyola University Chicago, Dr. Boyle, is an involved and direct
stakeholder of the program. The office of the Provost strives to promote and support an active
learning environment and more specifically takes on global initiatives on the campus. The Office
of International Programming is another direct stakeholder in the program. Its initiative
establishes Loyola as a diverse community and seeks out avenues to encourage global awareness
and establish programming and cultural exchanges that enhance understanding of diverse
cultures. They are important stakeholders because they are the key administrators that process
the paperwork, stay abreast with visa requirements as well as continue to refine and establish
innovative programs for International Students. Foreign sending agencies, meaning the
universities that send the students, act as a significant stakeholder as they are large supporters of
the program. Without their assistance and cooperation, recruiting students to participate in the
Chicago Center Program would be difficult for the Chicago staff. The professor of the Chicago
Program seminar plays a direct role in the pro grams success. Professor Michael Hines role is to
ensure that the students are engaging inside and outside the classroom. He, along with his
Teaching Assistant, facilitates outings outside of the classroom that help round out the hand-on
classroom experience.
Indirect Stakeholders
Along with direct stakeholders, there are also key indirect stakeholders that benefit from
the success of the Chicago Center Program. The Chicago Center Program operates in
conjunction with the U.S. Department of State. The program is designated by the Department of
State to issue documents needed for the students to obtain a visa to the United States. The
-
8/12/2019 ELPS 431, Evaluation in Higher Education
11/91
11EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
political culture in Washington D.C. heavily influences the operation and implementation of the
program. Without key stakeholders in Washington supporting cultural exchange programs, it
would be difficult for the Chicago Center Program to run effectively, if at all. Major funding at
the federal level is discussed in the Department of State sector that trickles down to the
Unive rsitys funding (Simpson, 2012).
The Loyola University Chicago community, mainly the students, is an indirect
stakeholder in the program. The main purpose of this program is a cultural exchange experience
in which the International Students are expected to further their knowledge and obtain a better
understanding of American culture. Without the students support, it would be d ifficult for the
International Students to have social interaction with the students and learn from each other. The
American students have the opportunity to interact with another culture if they choose to take
advantage of the opportunity.
Contextual Circumstances
There are several contextual events or circumstances that could affect the program that
might distort the evaluation. Since the program is highly regulated under the jurisdiction of the
U.S. Department of State, political contexts could change the way the program is evaluated.
Funding at the federal level shifts depending on who has authority (Simpson, 2012). Depending
on who is elected either in the House of Representatives or Senate greatly affects how much
money is granted for cultural exchange programs. Immigration laws also can affect the program
success. For example, as immigration becomes a more scrutinized topic in the U.S., immigration
laws and regulations may inhibit students from participating or make it financially impossible to
participate. Another contextual circumstance that could affect the evaluation is the social climate.
The program relies heavily on positive international relations in order to recruit International
-
8/12/2019 ELPS 431, Evaluation in Higher Education
12/91
12EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Students for the program. A main example of how this could negatively affect the program can
be seen in the recent news with Egypt. When the U.S. embassies closed in Egypt, hundreds of
International Students were unable to attend their visa interviews to come to the United States
causing many of them to cancel their programs in the United States. The University as a
collective also could affect the outcome of the evaluation. As global understanding is an integral
component of the mission at Loyola University Chicago, it is the duty of those in power to
uphold their mission and stay true to its values. Finally, budgeting at the University level can
have an effect on the success of the program. University money may be distributed differently
and may negatively affect the program if students have to pay more for activities such as
excursions around the United States.
Overriding questions
Processes: Are program participants actively engaging with the optional programs (i.e.
Host Student program & International House events) as well as the mandatory programs
(i.e. weekend field trips & excursions) as cultural experiences?
Outcomes: Are students developing (i.e. global awareness, multicultural competence and
self-efficacy) with the support of programming by the Chicago Center Program?
Logic Model
For this evaluation of the Chicago Center Program, the focus will be on the processes and
outcomes of programming offered through the Chicago Center Program. As noted in the Logic
Model (please refer to Appendix C), there are several inputs, outputs, outcomes, assumptions,
and expectations.
Inputs
-
8/12/2019 ELPS 431, Evaluation in Higher Education
13/91
13EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Time. For the current group of 30 students, the Chicago Center Program functions
smoothly with the Director and Graduate Assistant as the main staff members and the support of
the Vice Provost. The time that is dedicated to the program by staff is concentrated on
constantly improving and working on expanding the Chicago Center Program; the Graduate
Assistant dedicates time to planning the excursions and supervising the excursions with the
students, recruiting Loyola students for the Host Student Program, being available for questions
from the Chicago Center Program participants and duties related to the role as Teaching
Assistant for the required seminar course; the Director dedicates time to the program through
traveling, planning marketing strategies for expanding the program, supervising the Graduate
Assistant and is on-call for the students at all times.
Materials. The materials that are consumed by the Chicago Center Program include
marketing materials such as flyers and informational pamphlets to be handed out at recruiting
events. The Chicago Center Program is responsible for providing materials to students to
supplement their seminar class experience such as news articles and other class related
handouts. Participants consume various group meals during programming events such as
orientations and the excursion trips as well as the other budget resources allocated to the trips.
The Chicago Center Program requires a substantial budget to fund the programs including other
budget necessities such as: marketing materials and travel funding for the Director to describe
the co-curricular activities while recruiting students; tuition and stipend for the Graduate
Assistant who organizes the programs and trips.
Outputs
Programs. The main outputs of the Chicago Center Program are the programs created
specifically for its participants. The most appealing programs that assist in initial recruitment of
-
8/12/2019 ELPS 431, Evaluation in Higher Education
14/91
14EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
participants are the excursion trips to other major U.S. cities. Since these excursions only
happen two times each semester, the Chicago Center Program staff also organizes events
throughout the year on campus for various reasons. Programs such as pizza parties with Host
Students, trivia nights and holiday celebrations not only help the students to bond with each
other, but bring domestic students into the International House to expand their global experience.
The initiative of an international community is achieved as students from different cultures come
together under one roof.
Students. Students are cited as an output because they are the main participants intended
to reach in the Chicago Center programming initiatives. The 30 students from Brazil, China and
Korea make up the core group of students involved in programming. Student participation is also
important among Host Students and Loyola students in general. The Host Students are those
who voluntarily interact with the Chicago Center. The Host Students are the Americans most
likely to gain an expanded global perspective as a result of this program. They are self-selected
to support the Chicago Center Program students. Other Loyola students are likely to be reached
by the Program because there is a high chance that they will indirectly interact with Chicago
Center participants on campus through a shared class or mutual attendance at an event. It is an
intention that this will be a beneficial experience for all students involved, but understood that
not every interaction will be positive.
Outcomes
A few of the intended outcomes of the programming included in the Chicago Center
Program experience are improvement in English skills, general adaptability to a different culture,
improved global perspectives and marketability in alumni job search due to bilingual abilities
and cultural competence. These outcomes, along with personal growth, identity development
-
8/12/2019 ELPS 431, Evaluation in Higher Education
15/91
15EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
and improved academic records, have been cited in contemporary study abroad literature as the
desirable and most common outcomes of international education (Twombly et al., 2012).
Chicago Center stakeholders have cited the typical study abroad experience of U.S. students
abroad as inspiration for how the Chicago Center Program was designed and they intend for
program participants to have a similar experience as their American counterparts.
Assumptions
When evaluating the Chicago Center Program, there are several underlying assumptions
that need to be addressed. The first assumption of the Chicago Center Program is that all
program participants will partake in the excursion trips. Although highly encouraged, the
excursion trips are not mandatory and therefore it is unrealistic to assume that all participants
will attend. It is also assumed that all students will want to practice their English while making
friends who are native English speakers rather than only seeking friends in the International
House. Recent studies have discussed language acquisition as a key desire of a study abroad
program in a country where participants native language is not widely spoken. This is
considered the ideal way to acquire language skills and proficiency, but there has been
discussion around how simply studying abroad does not guarantee this acquisition; students need
to immerse themselves in the culture and possess the desire to practice (Twombly et al., 2012).
Another grand assumption is that Host Students will be receptive to Chicago Center
participants and welcome them into their culture. In order to develop a global perspective, a
desirable goal of most study abroad programs, students must demonstrate a sophisticated level of
self-efficacy in order to reach out to domestic students and build relationships. Although this
action is ideal, there are external factors that affect the ability for these exchanges to happen.
Twombly et al. (2012) describes a model of intercultural competence that is undoubtedly
-
8/12/2019 ELPS 431, Evaluation in Higher Education
16/91
16EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
influenced by the social context within which the interchange occurs (p. 71). This social
context essentially implies the idea that the Chicago Center participants need to experience a
welcoming environment in order to maximize their engagement with U.S. culture and language
development.
External Factors
There are several external factors that could inhibit these positive outcomes for the
Chicago Center Program. To begin, participants may have negative experiences with American
culture that could cause program participants to lose interest in engaging with the Loyola
campus. Secondly, participants may find a community consisting of speakers of their native
language which could hinder improving their English skills they had originally sought out to
improve.
Another external factor of the Chicago Center Program is the competition that exists in
the industry. International Students may have the desire to study in the United States as opposed
to other English speaking countries. Due to the constantly expanding market of education abroad
and the growth in opportunities in education around the world, students will be responsible in
deciding to study at the Chicago Center Program which is out of the control of the Chicago
Center Program stakeholders. Since there are many other countries that recognize English as an
official language, students have increased options to participant in programs in countries such as
Canada, England, Australia, as well as many other competing programs in the United States.
The Chicago Center Program could possibly falter if it does not use the proper tactics in
marketing to potential partner organizations for recruiting students and persuading them to
decide to send students to Chicago to study. The Chicago Center Program relies heavily on
receiving funding from the University to sustain its programming and expansion. The Programs
-
8/12/2019 ELPS 431, Evaluation in Higher Education
17/91
-
8/12/2019 ELPS 431, Evaluation in Higher Education
18/91
18EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
implement necessary changes in current practices. The participants are living together in the
newly formed International House. I t is also the Directors first semester in leading the program
and the staff has also adopted are more student focused with the addition of its first Graduate
Assistant from the Higher Education program. The staff members of the program are now more
focused on student issues, which previously have not been discussed in the past such as student
development and leadership.
Processes and Outcomes
Processes to be investigated. The processes that will be investigated through this
evaluation proposal will revolve around the topic of programming. The term programming in
the context of this proposal relates to activities that are organized by Chicago Center Program
staff and are funded by student tuition in addition to budgets provided by Loyola University
Chicago. Examples of programming include excursions to other cities, the weekend field trips
that supplement the seminar class, and activities with Host Students that are led by the program.
Additional processes that are to be investigated through this evaluation proposal are the learning
outcomes related to programming and overall attendance and satisfaction with the activities
planned by Chicago Center staff and faculty.
Rationale for investigating these processes. The rationale for investigating these
processes is to ensure that the Chicago Center Program is successful in effectively using its
budget for programming. It should reflect its own mission as well as the goals of the University
at the same time as engaging the students in co-curricular activities during their study abroad
program. Through investigating the processes the Chicago Center Program staff will be able to
further improve their strategies and current practices in planning programming events for
program participants and guarantee higher attendance and more meaningful programs.
-
8/12/2019 ELPS 431, Evaluation in Higher Education
19/91
19EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Outcomes to be investigated. Outcomes to be investigated by this evaluation plan will
revolve around learning and general student development. These outcomes may include
improvement of English language skills as well as improved intercultural communication, global
awareness, multicultural competence and a greater sense of community in the new context of a
university campus in the United States. Although the Chicago Center Program does not have a
defined set of learning goals, the outcomes described by the evaluators are designed to align with
both the mission of Loyola University Chicago and typical desired outcomes described in
contemporary study abroad literature ( Mission & Identity, 2013; Twombly et al., 2012 ).
Rationale for investigating these outcomes. The rationale for investigating these
outcomes of the Chicago Center Program is to ensure that the program is keeping abreast of
programming that will be interesting to students and unique enough in the contemporary field of
International Education so students desire to study at Loyola University as opposed to other
universities that offer similar programs. It is also vital to the success of future students that the
Chicago Center Program recognizes its diversifying population of program participants and is
conscious of the need to challenge students during their study abroad experience with the use of
programming as a tool in facilitating cultural exposure while creating connections between
students.
Strengths and Weaknesses
Establishing strengths and weakness is the key to provide our stakeholders with the
proper expectations. A strength of evaluating from a process perspective enables us to determine
how the program is being delivered and how successful it is being delivered. By looking at the
programming of the Chicago Center, we are able to determine if funding is appropriate. The
University may be very interested in exploring to ensure that the allocated funds are correct. An
-
8/12/2019 ELPS 431, Evaluation in Higher Education
20/91
20EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
inherent weakness is assuming that all students will participate in the programming. It also
assumes that each individual experience is the same and does not take into account each
individuals unique experience. It is difficult to say that every student will have the same
experience as his or her counterparts. Finally, another weakness exists in the length of time to
evaluation will take. Since the evaluation is centered on programming, it will be evaluated as
programs are occurring over the course of the entire academic calendar. If the evaluation is
conducted, the Chicago Center staff will be able to use in the future as evidence or support to
explain the importance of the programming and to what extent it meets the learning objectives.
Quantitative Methods
Participants
For the purposes of the study, the population studied will consist of the 30 Chicago
Center Program participants. The rationale for only choosing the students, as opposed to the
faculty and staff, is due to the fact that they have the most knowledge in terms of understanding
the experiences with targeted programming. The purpose of this current evaluation is to
understand the perceptions of the students and their response to programming provided by the
Chicago Center; the role of faculty and staff could potentially be evaluated in a separate study.
An advantage of using a small population size is that census-sampling frame can be used. Census
sampling allows for all 30 Chicago Center Program participants to be surveyed.
Survey design
Since the participants will not be randomly assigned to the group being studied, a quasi-
experimental design will be used. Census sampling will allow for all Chicago Center participants
to be surveyed thus eliminating the need for random assignment. In order to gain insight on the
processes and outcomes intended for the Chicago Center Program, a longitudinal research design
-
8/12/2019 ELPS 431, Evaluation in Higher Education
21/91
21EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
will be used to track the same participants over the course of the 16-week semester. In order to
measure change over the course of the semester, both a pre- and post- test will distributed to the
same participants at two different times. A pre-test post-test design will allow us to determine the
perceived ideas of the students prior to their engagement in programming and how they
experienced the intended outcomes in the post-test. For the study, a comparison group will not be
used, as it would not be feasible to include a comparison group due to a lack of resources at this
time.
Use of Pilot Study
A pilot test will not be conducted for several reasons. The small population size of 30
makes it difficult to select a decent population size to conduct a pilot study. In the pre-test, the
goal is to have participants answer with their instinct and authentic responses to the questions
provided in the survey. It would have a negative effect if the participants were to take the survey
twice. It is not possible to exclude the pilot-study participants because it would result in too
small a sample in the main study. Secondly, in order to capture a comprehensive shared
language, the questions are designed to be as specific as possible and incorporate definitions as
needed. Since English is not the native language for the participants, it is potentially unethical
and a threat to the students self -efficacy to select participants with the lowest English abilities to
take a pilot test to ensure that it makes sense. The evaluators do not intend to highlight lower
language abilities among certain participants versus others; choosing these participants could
negatively challenge their self-perception. Before the survey is officially released and ready for
implementation, the evaluators will consult direct program stakeholders such as the Chicago
Center Program staff and the Vice Provost for Academic Centers and Global Initiatives. This
-
8/12/2019 ELPS 431, Evaluation in Higher Education
22/91
22EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
will ensure that those stakeholders who work closely with the potential survey participants will
have the ability to offer their advice in regards to the language used and topics covered.
Instrument Description
The survey pre-test (Appendix D) contains 5 questions. The main question of the survey
contains seven statements measured on a Likert-scale (strongly agree, agree, neutral, disagree,
strongly disagree). The topics in the pre- test explore the students perception of world news,
ability to discuss cultural differences, personal values, and overall confidence in adapting to new
situations and were guided by a research study by Glass (2012). This research by Glass (2012)
was conducted on International Students studying in the U.S. and provided guiding topics for the
development of survey questions for this evaluation plan. For the purposes of this study,
additional details were included in the tailoring of questions to the Chicago Center Program in
order to accommodate the student population and level of English proficiency. The questions are
intended to gather the participants initial reactions of these statements prior to their exposure to
programming. The remaining four questions explore the student demographics.
The post-test (Appendix E) contains ten questions. The majority of the questions will be
measured on a Likert-scale (strongly agree, agree, neutral, disagree, strongly disagree). The
remaining questions, also guided by Glass (2012), are multiple-choice with a few questions
allowing the students to answer with more than one response. The post-test explores the same
topics as the pre-test, but also explores in depth students experiences with the programming.
The programs evaluated are the following: in-class assignments from the seminar that require
exploring Chicago, out-of-class field trips, International House programming, and excursions to
other major cities. There are multiple goals of the post-test. The main goal of the post-test is to
measure the development of the students at the conclusion of the Chicago Center Program
-
8/12/2019 ELPS 431, Evaluation in Higher Education
23/91
23EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
experience as well as their views on the United States. Since the Program is unique in the
amount of programming and excursions that are planned and led by Chicago Center Program
staff, the post-test will help in determining if participants are improving their personal
development and self-perception as global citizens. The other goals of the post-test are for the
evaluators to provide feedback to Chicago Center Program stakeholders in regards to the various
programming, such as the seminar field-trips and the International House events.
In the pre-test the demographics are asked at the end of the survey to ensure their answers
do not affect their responses to the survey questions. The students will also be asked their student
ID number at the beginning of both the pre-test and post-test to compare the responses of each
student. The pre-test should take no longer than 8-10 minutes and the post-test will take
approximately 15-20 minutes to complete. Due to the fact that the instrument was not piloted
with potential participants, the survey length was controlled by the approximate time it took each
evaluator to read and answer each question. The time was then doubled to estimate the time it
would take the program participants to complete the survey due to English not being their first
language as well as taking into account historical recall.
Implementation
The survey will be administered through the web. The pre-test will be distributed at the
Chicago Center Program mandatory orientation that takes place at the start of the each semester
in August and January. Since the orientation is mandatory and takes place in a computer lab, we
expect a 100% response rate on the pre-test survey. The post-test will be distributed at the final
in-class meeting of the seminar course when the participants present their final
presentations. Since there will be final presentations on the last day of class, there is an
anticipated response rate of 100% as all students have a graded oral presentation. Students will
-
8/12/2019 ELPS 431, Evaluation in Higher Education
24/91
24EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
be notified about the survey during the last class meeting ahead of time (see Appendix F) and
students will be encouraged to bring a tablet or laptop in order to complete the survey in class.
The Graduate Assistant will rent additional laptops for the participants who request one to utilize
for survey purposes on the last day of class. If for any reason a student does not participate in the
orientation or the final class, the survey will be distributed to them via e-mail for the opportunity
to fill out, but it will not be mandatory. They will be reminded that their responses will remain
anonymous and that the survey is voluntary.
The Graduate Assistant of the Chicago Center program will administer the survey both at
the orientation and the final class as well as act as the main contact for the survey. There will be
no incentives for the participants to complete the survey as the given response rate is quite high
regardless. The participants will be reminded at the pre-test survey that they will be invited to
take a similar survey at the end of the program in their seminar class.
Statistical Analysis
In order to analyze, code, and describe the data, SPSS will be used to shed light on the
distinct findings of the study. A test will be run for descriptive statistics on all of the variables
(frequencies for the nominal variables and means for the continuous ones). The nominal data will
include categorical values such as the demographics. As both the pre-test and post-test consist of
similar Likert scale questions, a paired samples t-test will be performed to examine how the
population has changed over time. This information will help further gage to what extent the
programming impacts the students perception of themselves, the U nited States, and how they
internalize cultural differences (refer to Appendices G and H for a survey construction maps of
the pre- and post-tests).
-
8/12/2019 ELPS 431, Evaluation in Higher Education
25/91
25EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
An independent samples t-test will be conducted to examine how the responses varied
amongst country of origin and gender. It is important for the evaluation to look at how the
programming is experienced by the three cultures represented: China, Brazil, and Korea. The test
may help determine cultural differences and their impact on how students experience
programming.
Presentation
The final results of the evaluation will be presented in the form of a table and then
eventually converted into a more visually pleasing chart. The seven statements will be placed on
the horizontal axis of a double bar graph and the vertical axis will be labeled one to five; one will
represent strongly disagree and five will represent strongly agree. The mean totals for each
statement will be illustrated from both the pre-test and the post-test. This design will be
completed to represent each country on its own graph. The demographic information will simply
be presented in the form of a pie chart whereas the other information collected in the evaluation
will be presented in the form of bar graphs with percentages in a column adjacent to the
horizontal bars; these graphs are automatically generated through the online survey builder,
Qualtrics, which will be used in this evaluation.
Qualitative Methods
Rationale
The qualitative approach will help the evaluators to gain a better understanding of the
students individual experiences with the programming in the Chicago Center Program. Focus
groups will allow for the collection of data that cannot be captured otherwise using the survey
method. As the overriding question focuses on the process and outcomes of the programming,
focus groups will humanize their experiences, which are difficult to qualify in statistical data and
-
8/12/2019 ELPS 431, Evaluation in Higher Education
26/91
26EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
provide more validity to their responses on the survey. The focus groups will also inform the
quantitative research done in the surveys. The focus groups will not address a topic distinct from
the survey but rather it will be used to understand why the participants responded in a certain
manner to the survey questions.
The focus groups will be instrumental in furthering the understanding of evaluating the
programs processes and desired outcomes (Wholey, et. al, 2010). Several questions are designed
around evaluating the processes of the Chicago Center Program. These questions consist of the
students reactions to the leadership present in the programs along with the programs activities
themselves. The questions designed around the outcomes focus on the students reactions to their
own personal development as a result of the program and the impact it had on their experiences
in the United States.
Focus Group Participants
All 30 students who are currently participating in the Chicago Center Program will be
invited via e-mail to the focus group interviews (refer to Appendix I). A criterion-based
sampling method will be utilized and the participants will be divided into separate groups based
on country of origin . Assuming that not all 30 students will attend the focus group, a 50%
interest rate is expected which could potentially create three focus groups to represent Korea,
China and Brazil. The rationale in dividing the students by country of origin is to provide them
with an optimal level of comfort in what they may consider to be an uncomfortable situation or
new experience. Common cultural background could also assist in historical recall when
students may be retrieving memories from times when their cultural identities were most salient
while in the United States; similar lived experiences may create more openness among the
participants and therefore in the focus group.
-
8/12/2019 ELPS 431, Evaluation in Higher Education
27/91
27EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
In the case that a drastically disproportionate amount of students respond to the call for
focus group participants (i.e. nine Chinese, four Koreans and one Brazilian), the graduate
assistant will be charged with the task of creating two focus groups based on varying levels of
English. Students will be equally distributed between groups by what the graduate assistant
believes is a variety in English skills to ensure that both groups will have students who tend to
talk more as well as students who are more reserved.
Protocol Instrument
The protocol instrument for the focus group portion of the Chicago Center Program
evaluation intends to learn more about the individual experiences of students during their
semester or year of study in the United States (refer to Appendix J). The focus group will begin
with an introduction to the evaluation plan, its goals and how the information gathered from the
interview will be used. The participants will be verbally reminded about the confidential nature
of a focus group and their ability to leave at any time. The consent form will reiterate the
confidentiality of the interview and mention that they can choose to leave at any point (refer to
Appendix K). Due to the fact that the focus group will have to take place around final
examinations, the participants will be promised that the focus group will last between 60 and 90
minutes. The promise of a meal will be emphasized in order to mitigate the feeling of losing
time for exam preparation; the focus group will be thought of as a conversation and library
break.
The questions will not be piloted with the students due to the small size of the group and
the general difficulty in recruiting students as focus group participants; the evaluators do not
want to risk losing participants for the interviews. In order to confirm the validity of the
questions and probes, the protocol instrument will be reviewed extensively with stakeholders
-
8/12/2019 ELPS 431, Evaluation in Higher Education
28/91
28EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
such as the Chicago Center Program Director, Graduate Assistant and the Vice Provost for
Academic Centers and Global Initiatives. The opinions of these stakeholders will help in
guiding the proper questions to be asked during the focus group. The focus group moderator will
have an additional meeting or two with the direct stakeholders to discuss strategies for a
successful interview. Staff members who have a close relationship with the program participants
may have insight for the moderator that will help with probing during the interview; they also
may be able to offer advice in regards to cultural nuances and actions or words for the moderator
to avoid in order to create a comfortable environment.
Implementation Procedures
The focus group will be conducted by Abbie Ray, one of the evaluators of the Chicago
Center Program. It will be advantageous to have Abbie conduct the interviews because of her
connection to this evaluation plan as well as her disconnection from the Chicago Center Program
participants. Since the students do not know Abbie they may feel more comfortable offering
critical opinions about the program, but will have the comfort of the group setting with peers
they have known for the semester. The focus group will take place toward the end of the
semester after the students have completed both excursion trips and are near the end of the
seminar course. To ensure comfort and convenience for the students, the focus group will take
place at the International House in the Directors office. The focus group will likely take place
over a weekend when the students have more open schedules, but in the case that there is a
weekday focus group, the Director has an alternative space for working during the 60-90 minute
focus group. The students will be informed that there will be a digital recording device in order
to capture all of their stories since there will only be one interviewer who will be guiding the
questions as well as managing notes on body language and group dynamics (refer to Appendix L
-
8/12/2019 ELPS 431, Evaluation in Higher Education
29/91
29EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
for note-taking sheets to be used by the moderator). Students will be provided with a meal from
a restaurant representing their country of origin such as Brazilian Bowl or DAK Korean Chicken
Wings.
Focus Group Analysis
In order to analyze the data from the focus groups, we will transcribe the focus group and
construct a map to develop a coding scheme prior to the interview. The Graduate Assistant will
be tasked with transcribing the data as it is a part of the job description. The major constructs that
we will be evaluating are the following: diversity (DIV), personal development (PDEV), global
awareness (GLAW), multicultural competence (MULTC), programming experiences
(PROGEX), challenges (CHAL), efficacy (EFCY), and community (COMM) [refer to Appendix
M]. Additional constructs may be added if new emerging themes develop in the responses of the
participants. Once all of the focus group interviews are completed and transcribed, the
evaluators will closely review one transcription to compare the coding consistency and further
refine the a priori codes if necessary. In order to avoid biases in selecting a transcription to
review, it will be decided prior to the review that the shortest interview will be selected. To
ensure inter-rater reliability, the evaluators will meet prior to the focus group to discusses the a
priori codes and develop a shared understanding of each of the constructs.
A cutting/sorting technique will be used to analyze the data. The cutting/sorting
technique will allow us to identify and see the themes across the data. Member checking will be
used at the end of the study to ensure that the evaluators are interpreting the data in a way that
captures the voices of the participants. Students will have an opportunity to review the overall
themes described by the evaluators who code and interpret the data (Creswell, 2009). The
Graduate Assistant will have a large role in the member checking portion of the study as she will
-
8/12/2019 ELPS 431, Evaluation in Higher Education
30/91
30EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
be the point person for contacting students who have already returned to their home countries by
the time the data has been prepared for this process. The data will be triangulated with the
quantitative data as the study consists of two methods of data collection. Triangulation will
consist of comparing the findings from the survey as a way of informing the coded data from the
focus group interviews. Through the process of triangulating, the evaluators will have the ability
to check the focus group participants specific stories against general themes from the survey. It
will help decide if their accounts are unique events or generally agreed upon.
Limitations and Personal Bias
With Abbie as the facilitator of the focus group, it will eliminate personal bias as she
does not play a role in the Chicago Center Program. On the other hand, it may also serve as a
limitation as the students are unsure of who she is and may feel uncomfortable disclosing their
feelings and reactions to a stranger. Another limitation of the method is the probability that group
think might occur. Group think is when a group of people seek a census and agreement often
eliminating the opportunity for alternative ideas and opinions. As a way to counteract group
think, it is imperative that we make the participants aware that the purpose of the focus group is
purely evaluation and we encourage every opinion because it will help make the program
stronger and more effective for future students. A final limitation is cultural differences.
Participants in the study may not provide us with the most authentic answers because it may not
be culturally acceptable in their country to engage in conflict and disagreement. It is important
that the facilitator makes it clear that no statement or question will be diminished or offense and
reiterate that they are in safe space to speak their mind.
Results Presentation
-
8/12/2019 ELPS 431, Evaluation in Higher Education
31/91
31EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
The results will be presented through various sources. First, a word cloud will be
generated through NVIVO to demonstrate the most common words used throughout the
interview. This method will provide a visual resource for stakeholders to review common themes
among Chicago Center Program participants and discuss topics that are most important to them.
Information will also be presented by using direct quotations from the students to help capture
their voice as a supplement to the findings from the quantitative survey; this will personalize the
data is presented to stakeholders as a way to use stories to enrich the evaluation process. About
two to three brief stories from students, more if time permits, will be highlighted during the
results presentation to the stakeholders for purposes of identifying patterns in the data, infusing
life into the quantitative data where there are correlations between the sets as well as encourage
conversation between evaluators and stakeholders (Kreuger, n.d.).
Limitations
There are a few limitations to the overall evaluation plan of the Chicago Center Program
that the evaluators have attempted to address through explicit details and examples in the
quantitative design and with face-to-face explanations of the process as often as possible. The
participant group from whom the evaluators will collect their entire data set identifies as
International and English is none of their first language. The students have varying levels of
proficiency with the language and some have become more adapted to culture in the U.S. than
others, resulting in a wide range of interpretation and understanding of the evaluation.
A strong limitation to implementing an evaluation plan during this academic year is in
regards to a general dissatisfaction with the International House. Through conversations with
Chicago Center Staff, the evaluators have learned about the list of issues related to the physical
building and dissatisfaction of program participants. Since the building was an incomplete
-
8/12/2019 ELPS 431, Evaluation in Higher Education
32/91
32EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
renovation project at the time of arrival for the Chicago Center participants, there is a great
possibility that their views on the U.S. university system is not as positive as it may have been
for past participants who lived in dormitories with domestic students. This building can also
create an isolating effect on the students that may have caused them to rely on befriending their
peers in the Chicago Center Program, rather than taking the risk in reaching out to domestic
students.
Next Steps
The Chicago Center Program at Loyola University has become well established over its
past five semesters but has much room for continual growth and improvement. This evaluation
plan is another step in the direction of improvement that will ideally be used again in subsequent
years. Important steps in the immediate future will include reconnecting with program
participants after they return to their home countries and they have had time to reflect on the
effects of reverse culture shock. Students may not have realized the impact of the various
programming efforts such as the seminar course field trips or their time spent at events with Host
Students; conversations after they are taken out of the environment of Chicago may illustrate a
different set of thoughts. In the coming years after the Chicago Center Program has established
itself more and reached out to a wider audience of students from other countries, a re-designed
study that includes International Students at Loyola who are not in the Program may provide
stakeholders with additional insight.
Stakeholders and evaluators could potentially draw connections between the Chicago
Center Program and International Students at Loyola because of the complete assimilation
process that the non-Program participants experience without the additional support of
programming and staff members outside of their visa advising. These ideas for improving the
-
8/12/2019 ELPS 431, Evaluation in Higher Education
33/91
33EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
evaluation in future semesters along with other recommendations will be presented to
stakeholders and Chicago Center staff during the initial presentation of data by the evaluators. A
defined list of recommendations combined with the open-mindedness of the evaluators for
conversation around ideas for the Program will result in a rich plan for the Chicago Center in the
coming years.
-
8/12/2019 ELPS 431, Evaluation in Higher Education
34/91
34EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
References
Mission and Identity. (2013) Retrieved Sunday, September 29 th, 2013 from
http://www.luc.edu/mission/mission_vision.shtml
Creswell, J.W. (2009). Research design: Qualitative, quantitative, and mixed method approaches
(3rd Edition). Thousand Oaks, CA: Sage Publications.
Garanzini, M. (2013, September). Loyola at a Glance. http://luc.edu/keyfacts/index.shtml
Retrieved Sunday, September 29 th, 2013, from http://luc.edu/keyfacts/index.shtml
Glass, C. R. (2012). Educational experiences associated with international students learning,
development, and positive perceptions of campus climate. Journal of Studies in
International Education, 16 (3) 228-251. doi: 10.1177/1028315311426783.
Institute of International Education. (2013). Open doors data: Fast facts. Retrieved from:
http://www.iie.org/Research-and-Publications/Open-Doors/Data/Fast-Facts
Krueger, R. A. (n.d.). Using stories in evaluation. In J. S. Wholey, H. P. Hatry & K. E.
Newcomber (Eds.), Handbook of practical program evaluation (Third Edition). (404-
423). Jossey-Bass: San Francisco.
Salisbury, M., Umbach, P., Paulsen, M., & Pascarella, E. (2009). Going Global: Understanding
the Choice Process of the Intent to Study Abroad. Research In Higher Education , 50(2),
119-143. doi:10.1007/s11162-008-9111-x
Simpson, D. (2012). International Exchange Programs Strengthen the Global Community. Public
Manager , 26-28.
http://www.luc.edu/mission/mission_vision.shtmlhttp://www.luc.edu/mission/mission_vision.shtmlhttp://luc.edu/keyfacts/index.shtmlhttp://luc.edu/keyfacts/index.shtmlhttp://luc.edu/keyfacts/index.shtmlhttp://luc.edu/keyfacts/index.shtmlhttp://luc.edu/keyfacts/index.shtmlhttp://luc.edu/keyfacts/index.shtmlhttp://www.iie.org/Research-and-Publications/Open-Doors/Data/Fast-Factshttp://www.iie.org/Research-and-Publications/Open-Doors/Data/Fast-Factshttp://www.iie.org/Research-and-Publications/Open-Doors/Data/Fast-Factshttp://luc.edu/keyfacts/index.shtmlhttp://luc.edu/keyfacts/index.shtmlhttp://www.luc.edu/mission/mission_vision.shtml -
8/12/2019 ELPS 431, Evaluation in Higher Education
35/91
35EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Twombly, S. B., Salisbury, M. H., Tumanut, S. D., and Klute, P. (2012). Study Abroad in a New
Global Century: Renewing the Promise, Refining the Purpose. New Jersey: Wiley
Periodicals, Inc.
Watson, J. R., Siska, P., & Wolfel, R. L. (2013). Assessing Gains in Language Proficiency,
Cross-Cultural Competence, and Regional Awareness During Study Abroad: A
Preliminary Study. Foreign Language Annals , 46 (1), 62-79. doi:10.1111/flan.12016
Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (Eds.) (2010). Handbook of practical program
evaluation (Third Edition) . Jossey-Bass: San Francisco.
-
8/12/2019 ELPS 431, Evaluation in Higher Education
36/91
36EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Appendix A: Previous Evaluations
Chicago Center Program: Program Evaluation Spring 2012
20 participants
Using a multiple choice scheme, mixed with open-ended questions, students were asked a seriesof questions.
Section: Housing
1) Which building did you live in this semester?Sovereign Apartmnet (1) Spring Hill (1) Fordham (1) Santa Clara (1)Marquette (1) Rockhurst (5) Fairfield (5) Le Moyne Hall (5)
2) Did you have an American roommate or another student from the Chicago Center Program?a. American roommate (17)b. Other Chicago Center student (2)no answer (1)
3) Were the housing accommodations provided clean and comfortable?a. Exceptional (6)b. Good (12)c. Fair (2)d. Poor
4) Were the housing accommodations in a convenient location (close to classes and activities)?a. convenient (18)b. Not convenient (2)Comments: (12 no comments) Of those who did comment, campus construction seemed to be an
issue. One also mentioned they liked the shuttle service provided.
5. How would you rate the housing experience overall?a. Exceptional (8)b. Good (11)
c. Fair (1)d. Poor (0)Comments: 13 did not comment. Of those who did, 4 said it was great, comfortable, and theview is wonderful. One said there was not heat. Another 2 said their roommates were wonderful.
Section: Social Life
6. Did you feel you were well informed about social activities on campus? (sports, clubs, activities
-
8/12/2019 ELPS 431, Evaluation in Higher Education
37/91
37EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
etc)?a. Well informed (18)b. Not well informed (2)
7. Did you feel you given opportunities to participate in social activities with American students?a. Given many opportunities (6)c. Given some opportunities (13)d. Given a few or no opportunities (1)
8. With whom did you spend the most free-time with this semester?a. Other Chicago Center students (11)b. My roommate (6)c. My host student (0)d. Other (for example an American friend you made in class) (5)Comments: Some suggested more activities involving American students, while one student simplysaid they oftn went out alone. A few commented that they liked their roommate and host student
9. Where did you spend the most free-time this semester?a. On campus (11)b. Off campus in Rogers Park (0)c. Off campus in other neighborhoods (9)
10. Were there any challenges that hindered you from participating in social activities on campus?If so, please explain.12 studnts left this comment blank or answered no. Of the remaining 8 students, 2 creditedthe language barrier as an impedment, and 2 others said that cultural differences were to blame forlack of social involvment. One mentioned they were simply treated as a guest the wholetime, not allowing for deep conversation. Another said they were not treated well by "foreigners"
11. How important was your roommate in creating a positive social experience?a. Very useful (5)
b. useful (9)c. Not very useful (3)d. Not useful at all (3)
12. How important was your host student in creating a positive social experience?a. Very usefulb. useful
-
8/12/2019 ELPS 431, Evaluation in Higher Education
38/91
38EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
c. Not very usefuld. Not useful at all
Section: Academics
13. Did you feel that you participated in an interesting and challenging range of courses?a. Yes, I participated in a wide range of course (19)b. No, I think the course options were limited (1)Comments: Some of the comments expressed a desire to see more courses offered. However, therewerewere several students whose selection was limited by their institution in China.
14. Did you feel that the number of courses and amount of course work were appropriate?
a. Yes, the course work was appropriate (20)b. No, the course work was inappropriate (either too demanding or too easy) (0)Comments: 2 students wanted more courses
15. What was your favorite course this semester? Why?Answers varied from one student to the next However, quite a few students liked classes involving langauges.
16. What was your least favorite course this semester? Why?Answers varied from one student to the next However, Marketing and Business courses s eemed
to be the students' least favroites. They simply described them as difficult.
17. Are there additional courses that you think we should try to incorporate into the program?If so, please describe.
Section: Overall Experience
18. How would you rate the Chicago Center Program Overall?a. Excellent (18)b. Good (1)
c. Faird. Poorno answer (1)
19. What about the program do you think could be improved in the future?Add more activities that aren't museums, more diversity and interaction with students fromother cultures. The program should be longer than just a semester.
-
8/12/2019 ELPS 431, Evaluation in Higher Education
39/91
39EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
20. What advice would you give to students from your school who are thinking of applying to theChicago Center Program?Enjoy the freedom you have now and take an adventure, make sure to get an American roommateIt's a wonderful opportunity to improve your English, apply ASAP, it's a cool program, great chanceto travel and explore a new country and culture.
-
8/12/2019 ELPS 431, Evaluation in Higher Education
40/91
40EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Chicago Center Program: Program Evaluation Fall 2012
18 participants
Using a four-point likert scale, participants were asked to rate a series of questions.In addition, participants were also asked to comment on a few questions.(questions 11, 14, 18, 19, 20 are open-ended questions)
completely disagreeslightly disagreeagreestrongly agreeno answer
Section: Housing
1. I was provided with all of the necessary materials (pillows, sheets, etc) upon arrivalcompletely disagree (0) agree (4) (22%) no answer (0)slightly disagree (1) (6%) strongly agree (13) (72%)
2. Housing was in a convenient location close to transportation and classescompletely disagree (0) agree (4) (22%) no answer (0)slightly disagree (0) strongly agree (14) (78%)
3. Housing was clean and comfortable
completely disagree (0) agree (3) (17%) no answer (0)slightly disagree (0) strongly agree (15) (83%)
4. I felt safe in my dorm and the surrounding neighborhoodcompletely disagree (0) agree (2) (11%) no answer (0)slightly disagree (0) strongly agree (16) (89%)
5. Housing allowed me to interact with students from diverse backgrounds(not just Chicago Center students)completely disagree (0) agree (5) (28%) no answer (0)
slightly disagree (4) (22%) strongly agree (9) (50%)
Section: Social Life
6. I was well informed about social activitiescompletely disagree (0) agree (7) (39%) no answer (0)slightly disagree (2) (11%) strongly agree (9) (50%)
-
8/12/2019 ELPS 431, Evaluation in Higher Education
41/91
41EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
7. Many opportunities were provided to socialize with American studentscompletely disagree (0) agree (9) (50%) no answer (0)slightly disagree (2) (11%) strongly agree (7) (39%)
8. My host student communicated well with me and was available to answer questions and offeradvicecompletely disagree (4) (22%) agree (1) (6%) no answer (0)slightly disagree (4) (22%) strongly agree (9) (50%)
9. My host student was an important resource for me throughout the semestercompletely disagree (4) (22%) agree (2) 11%) no answer (0)slightly disagree (4) (22%) strongly agree (8) (45%)
10. I bonded (became close) with the other members of my programcompletely disagree (0) agree (5) (28%) no answer (0)slightly disagree (0) strongly agree (13) (72%)
Section: Academics
12. The courses I was able to take fit my interests and goalscompletely disagree (0) agree (6) (33%) no answer (0)slightly disagree (3) (17%) strongly agree (9) (50%)
13. The work required of me in my courses challenged me as a studentcompletely disagree (0) agree (9) (50%) no answer (0)slightly disagree (0) strongly agree (9) (50%)
15. (no question is provided)
completely disagree (0) agree (3) (17%)no answer (2)(11%)
slightly disagree (0) strongly agree (13) (72%)
Section: Overall Experience
16. Overall the Chicago Center Program was a positive experience for mecompletely disagree (0) agree (3) (17%) no answer (0)slightly disagree (0) strongly agree (15) (83%)
17. I would recommend the Chicago Center Program to otherscompletely disagree (0) agree (2) (11%) no answer (0)
-
8/12/2019 ELPS 431, Evaluation in Higher Education
42/91
42EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
slightly disagree (0) strongly agree (16) (89%)
Section: open-ended questions
11. Were there places in Chicago that you wanted to go to but could not because of transportationor cost?5 participants answered no, 3 mentioned Chicago Bulls game, 2 mentioned the Art Institute,2 mentioned the University of Chicago, and 2 wanted to go to outlet
14. What course did you gain the most from this semester? Which course was did you gain theleast from? Explain each.Is participants did not distinguish between which they gained the most from and which they gainedthe least from, then their answers were ommitted from this evaluation.
Most - International Relations (mentioned 2), Dramatic/Theatrical Process, Management 320,Operations Management, Microbiology, Advanced Speaking and Listening.Why - To learn more about the relationship China and America share, writing improved,learned about American management
Least - The Chicago Experience, Organizational Change and Development, Advanced Listeningand Speaking, Intro to PharmacologyWhy - We can learn more history
18. What part(s) of the Chicago Center Program did you feel were the most enjoyable andinteresting. Explain.
Campus life, meeting new friends, campus resources, kitchen events, and 12 participantsmentioned they liked the field trips.
19. What part(s) of the Chicago Center Program did you feel could be improved for nextsemester? What specific changed should be made?
More cultural parties throughout the semester, don't go to DC, advisers, group dinners withperformances, nice professors, program should be more than one semester, there should not be a tripas soon as students arrive in-country, more course options, and 2 participants mentioned that classesshould be well-planned, and another 2 mentioned that living with an American roommate should beanoption.
20. For the Chicago Center Program, excursion trips play a major role. What was your favorite part(s)
-
8/12/2019 ELPS 431, Evaluation in Higher Education
43/91
43EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
of the trips? What was your least favorite part(s)? How could the trips be improved in the future?
2 participants thought the group dinners were good, would like to visit other universities, 3 did notenjoy the museums, 4 did enjoy the museums, 1 wanted more time to explore the city, 2 participantswant more to be given more options, some felt the trips were arranged well, 2 wanted morepreparation before the first trip, makes the trips longer was suggested, and some felt there was toomuch walking.
Additional Comments:1 student mentioned how they would like more freedom in selecting courses.
-
8/12/2019 ELPS 431, Evaluation in Higher Education
44/91
44EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Appendix B: Exploring the United States through Chicago , Course Documents
LOYOLA UNIVERSITY CHICAGO FALL TERM 2013UNIV 110.001: Exploring the United States through Chicago
Location: Room 423, Corboy Law Center, Water Tower Campus Time: Friday 4:15p.m.-6:30p.m.
Instructor: Michael Hines T.A.: Katie StephensPhone: (815) 275-7834 Phone: (314) 467-1814
Email: [email protected] Email:[email protected]
Office Hours: by appointment
Course DescriptionThe city of Chicago is a place of diverse cultures, rich history, and vibrant urban life. Using the city as a lens, thiscourse seeks to explore a wide range of contemporary American issues, including politics, history, race, class, andculture by examining how they play out in the Windy City . Chicago Center students will strengthen theirspeaking, writing, analysis, and presentation skills through the course content, which includes formal lectures, guestspeakers, and a number of outings that allow us to move beyond the campus and explore the sights and sounds ofChicago.
Learning OutcomesThe following represent the objectives or learning outcomes of the course:
Students will identify key events, people, and places that have impacted the growth and development ofthe city of Chicago through formal assessments.
Students will strengthen their spoken English skills by presenting a short analysis of an article from alocal news outlet.
Students will strengthen their written English skills through journaling, short response questions, and posting comments to course articles throughout the semester.
Students will apply basic research techniques in order to complete a collaborative project focusing on thehistory and culture of one of Chicagos historic neighborhoods.
Students will display an attitude of open mindedness and cultural sensitivity by working in cross culturalgroups throughout the semester.
Required Materials Please bring writing utensils and a notebook to every class, as well as a folder to keep any important informationyou receive. There are no required texts, as all course readings will be made available via Sakai, but it i s theresponsibil ity of the student to use Sakai to manage and print these materi als.
Assignments
Students will be graded on a number of different assessments that seek to build their skills in writing, speaking,analysis, and presentation.
Journal Entries: 20%In the News Presentation: 10% Midterm Assignment: 15%Chicago Neighborhood Project: 45%Participation: 15%Total: 100%
mailto:[email protected]:[email protected] -
8/12/2019 ELPS 431, Evaluation in Higher Education
45/91
45EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Assignment Descriptions
Journal Entries: An important part of making the most out of the semester and getting a great grade in this coursewill be writing journal entries about the topics covered in class, readings, or class outings. These entries will be ondifferent topics and have prompts or questions for you to answer, but there are some general expectations to followfor each. You will be expected to type the journal entries and submit them online via the Sakai site for the course
(you will receive instructions on how to submit work during the first class). Each entry should be a minimum of 2pages, Ti mes New Roman font, size 12, and dou ble spaced . Journal entries will always be graded and returned thenext class meeting so that you can easily keep track of your grade. An excellent journal entry will do the following:
Answer each question clearly and completely Adhere to the required length, font, and spacing Be written using good grammar and English conventions *to the best of the writers ability*
In the News Presentation: An individual presentation. Each student will sign up to bring in an article related to a particular topic covered in class. Students will explain the content of the article, and answer questions from the classabout the topic.
Midterm Assignment: An essay prompt based on the material covered in class during the first half of the semester.The midterm will be assigned on Oct. 25 and due Nov. 8.
Chicago Neighborhood Project / Presentation: Group project group centering on a neighborhood in Chicago.Groups will create a short research paper describing the neighborhood, a video guide showcasing important placeswithin the neighborhood, and a group presentation to give to the class.
Participation: Composite score based on attendance, timeliness, and willingness to take part in class discussionsand activities.
Grading ScaleAssignments will be graded based on the following scale:
A= 92-100B= 84-91C= 76-83D= 67-76F= 66-below
Class Expectations
Students are required to complete all assignments by the due date. Late assignments will be assessed a penalty ofone letter grade per day unless otherwise arranged with the instructor.
Attendance is expected at every class. If you are unable to attend a class meeting or outing for any reason, it is yourresponsibility to notify the instructor beforehand. Missing 2 or more class meetings or outings will negatively affectyour grade for the course.
Since this course is built on forming a community of learners, your participation and input are absolutely key toyour learning as well as the success of the class as a whole. You are expected to come to class each week preparedwith any readings or assignments, and ready to take an active part in class and small group discussions. In order tocreate an environment where everyone s voice can be appreciated and heard, cell phones must be silencedand away, and laptops must be closed during class unless used specifically for note taking.
-
8/12/2019 ELPS 431, Evaluation in Higher Education
46/91
46EVALUATION PROPOSAL: CHICAGO CENTER PROGRAM
Academic HonestyAcademic honesty is an expression of the responsibility of Loyola University faculty, students, and staff, to pursueknowledge with sincerity, honesty, and integrity. Plagiarism, presenting someone elses words, images, or ideas,WITHOUT giving credit, will not be tolerated. A s tudents failure to practice academic honesty, depending on theseverity of the misconduct, will result in sanctions ranging from the grade of F for the assignment to expulsion fromthe program.
For specific policies and procedures, see the following website:http://luc.edu/education/academics_policies_integrity.shtml
ScheduleThe following is a schedule for each week with important assignments and locations. Any changes to this schedulewill be communicated through email and sent far in advance of the class date.
Aug. 30 th: Introduction to Course / Chicagos History
Sept. 7 th: F ield Trip: Chicago Hi story M useum
Sept. 13 th: Diversity and Ethnicity in Chicago Neighborhoods
Chicago Neighborhood Project explained and groups assigned.
Sept. 21 th: Fi eld Tri p: National Mu seum of M exican Art
Sept. 27 th: Chicago Poems and Chicago Blues: Music and Literature in Chicago
Sept 28 th: F ield Trip: Blues Guitar U nsung: Old-town School of Folk M usic
Oct. 4 th: NO CLASS Excursion Tr ip to: L.A.
Oct. 11 th: Politics and Change: The National Scene
Oct. 18 th: Politics and Change: The Local Level *
Oct. 25 th: Art and Appetite: How Food and Culture Shape Chicago
Midterm Assigned Nov. 2 nd: F ield Trip: The Art I nstitute of Chi cago
Nov. 8 th: The Future of Chicago Pt. 1: Education and Young People
Midterm Due
Nov. 16 th: TF A H eadquarters Chicago
Nov. 22 nd: Future of Chicago Pt. II: Environment , Technology, and Innovation
Nov. 29 th: NO CL ASS: T hanksgiving Br eak
Dec. 6 th: Chicago Neighborhood Project Presentations
Chicago Neighborhood Project and Presentations DueDec. 13 th: Finishing Chicago Neighborhood Project Presentations (if needed)
http://luc.edu/education/academics_policies_integrity.shtmlhttp://luc.edu/education/academics_policies_integrity.shtmlhttp://luc.edu/education/academics_policies_integrity.shtml