Evaluation & Assessment Final Project

Summer Enrichment at Loyola (SEaL) Social Skills Development Evaluation

Loyola University Chicago December 2014

Antoaneta Topalova and Laura Roman

SEaL EVALUATION PLAN 2

SEaL Evaluation Plan

Antoaneta Topalova and Laura Roman

Loyola University Chicago

ELPS 431


TABLE OF CONTENTS

EVALUATION PLAN INTRODUCTION .................................................................................. 5

SEAL PROGRAM OVERVIEW ..................................................................................................... 5

SIGNIFICANCE OF PROBLEM ............................................................................................................. 6

CONTEXT AND HISTORY OF THE PROGRAM .............................................................................. 6

RICH DESCRIPTION OF THE PROGRAM ........................................................................................ 7

STAKEHOLDERS ................................................................................................................................... 8

REVIEW OF LITERATURE .......................................................................................................... 9

LOGIC MODEL .................................................................................................................................. 10

EVALUATION APPROACH ........................................................................................................ 11

QUANTITATIVE APPROACH ................................................................................................... 12

SURVEY INSTRUMENT ....................................................................................................................... 13

SURVEY METHOD & DESIGN ......................................................................................................... 14

SURVEY POPULATION AND SAMPLING PROCEDURE ................................................................ 15

STATISTICAL ANALYSIS ................................................................................................................... 16

QUANTITATIVE RESULTS PRESENTATION ................................................................................. 17

QUALITATIVE APPROACH .................................................................................... 17

QUALITATIVE QUESTIONS .............................................................................................................. 18

FOCUS GROUP PARTICIPANTS ....................................................................................................... 18

FOCUS GROUP PROCEDURE & IMPLEMENTATION ................................................................. 19

MODERATOR ....................................................................................................................................... 22

QUALITATIVE RESULTS PRESENTATION ................................................................................... 22

ANALYSIS PLAN ............................................................................................................................... 22

LIMITATIONS ................................................................................................................................... 23

TIMELINE ............................................................................................................................................. 24

BUDGET .................................................................................................................................................. 25

NEXT STEPS ....................................................................................................................................... 25

REFERENCES ..................................................................................................................................... 27

APPENDICES

APPENDIX A: MANUAL &STUDENT ASSESSMENT REPORTS 2014 ...................................... 29

APPENDIX B: MAP OF PARTNERING SCHOOLS .......................................................................... 62

APPENDIX C: LOGIC MODEL.......................................................................................................... 63

APPENDIX D1: PRE-SURVEY EMAIL INVITATION .................................................................... 64

APPENDIX D2: POST-SURVEY EMAIL INVITATION ................................................................. 65

APPENDIX E1: PRE-SURVEY PERMISSION FORM .................................................................... 66

APPENDIX E2: POST-SURVEY PERMISSION FORM .................................................................. 67


APPENDIX F: SUMMER 2014 SEAL ASSESSMENT .................................................................... 68

APPENDIX G: SURVEY CONSTRUCT MAP ................................................................................... 77

APPENDIX H: SURVEY QUESTIONNAIRE- EVALUATION PLAN ........................................... 80

APPENDIX I: FOCUS GROUP INTERVIEW QUESTIONS ............................................................ 85

APPENDIX J: FOCUS GROUP PROTOCOL .................................................................................... 86

APPENDIX K: FOCUS GROUP RECRUITMENT MATRIX .......................................................... 88

APPENDIX L: PARENTAL OR LEGAL GUARDIAN PERMISSION FORM ............................... 89

APPENDIX M: CODING RUBRIC ..................................................................................................... 91

APPENDIX N: TERMS OF PARTICIPATION................................................................................... 92

APPENDIX O: TIMELINE ................................................................................................................... 93

APPENDIX P: BUDGET ...................................................................................................................... 94

APPENDIX Q: PRESENTATION ........................................................................................................ 95


Evaluation Plan Introduction

The objective of this evaluation plan is to assess to what extent the SEaL program is

meeting its goal to further develop students social skills. As the reader endeavors in this

evaluation plan, they will encounter research findings supporting the limited research on social

skills development among youth afterschool programs. Durlak and Weissburg (2007) emphasize

how more time and resources seem to be invested in research pertaining to students academic

development, and this is why we have decided to orient our focus to social skills development

among youth. According to Jones and Bouffard (2012) academic development is as important as

social skills development.

SEaL Program Overview

The Summer Enrichment at Loyola (SEaL) program was started in 2011 at Loyola

University Chicago as a service to the Chicago Area community. The program was intended to

increase college access among mid-performing high school students identified as low-income,

first-generation, and/or students of color (Manual & Student Assessment Reports 2014; see

Appendix A). The SEaL program offers three weeks of skill development in key academic

subjects such as math, science, reading, and writing. Along with training on college related

topics such as financial aid and college applications; additionally, the SEaL program offers all

students LifeSkills workshops led by Loyola college students where the students learn about

college life and college preparation. Because this program is intended to serve families of low-

income, it is a free program to all students and families (Manual & Student Assessment Reports

2014; see Appendix A). This evaluation plan will focus on assessing the social skills

development of the students in the program. We have defined social skills to be in the form of

these activities and interactions: peer to peer interaction, public presentations, team work,


cultural competencies, interpersonal skills (i.e. engaging in activities with others),

communicating with others/networking, and reaching out for help.

The general evaluation approach of this assessment plan will be outcomes based with a

formative judgment. The intention is to determine how the social component of the program can

be improved. The focus of this evaluation plan is to assess the short, medium and long-term

social outcomes of the SEaL program, which are highlighted in our logic model (see Appendix

C). The use of Fitzpatricks, Sanders and Worthens (2003) formative, with an outcomes based,

question will be taken into account; how can we revise our curricula to better achieve desired

outcomes? (p. 27).

Significance of Problem

Our intention is to see how this program influences the students social skills development and

whether the program meets its social development objectives. In previous years SEaL program

administrators have gathered survey data primarily on academic skills development, yet limited

data on social skills development. We are constructing an evaluation of the social aspect of the

SEaL program because research pertaining to academic skills development seems to dominate

over research related to social skills development (Durlak & Weissburg, 2007).

Context and History of the Program

The SEaL program, which is designed for mid-performing students, low-income, and students of

color, is held at Loyola University Chicago (LUC) Lakeshore Campus (LSC). Mid-performing

students are defined by the high school teachers as students who need extra academic guidance

and support (K. Chavez, personal communication, October 3, 2014). The SEaL program at LUC

began in 2011 with nine students and has grown tremendously, from 60 students in 2013 to 135

students in 2014 (K. Chavez, personal communication, October 3, 2014). As a disclaimer, we


would like to note that throughout this evaluation plan we will be referencing data and

information from the 2014 summer cohort. However, the assessment part of this plan is

designed to be taken by future summer cohorts, since we anticipate that the enrollment numbers

will not exceed 160.

Summer 2014 was the first year that the SEaL program included rising 9th grade

students. Students were recruited from all parts of the city in order to increase diversity and

build a sense of community amongst them. The participating high schools from the Chicago

land area were selected because they were located in low-income urban communities (see

Appendix B).

Rich Description of the Program

High school counselors were asked to nominate students for the program in early March of

2014 and the students were asked to apply by early May of 2014. The nomination process

consisted of the counselors providing students contact information to SEaL. Students who fit

the earlier mentioned admission requirements were invited to apply for the program by SEaL

staff. Admitted students were responsible for taking public transportation to and from the

campus with the public transportation fare cards provided by the program. The participants

commuted to campus Monday thru Friday from 8:30AM- 5:00PM. Since this is a voluntary

program the students were asked to sign a Terms of Participation form stating that they

commit to attend every day the program is in session (see Appendix N).

The program was in session during the summer for three weeks, during which students

engage in daily activities purposefully designed to develop their academic and social skills. The

students participated in recreational activities along with academic workshops. They were also

allotted a generous amount of time for breakfast and lunch, during which they could engage in


relationship building dialogues (see Appendix A).

Stakeholders

The program was governed by several stakeholders, some were direct and others were indirect.

Direct stakeholders included the assistant director of the Summer Scholars program, the graduate

program coordinators (GPCs), the college coaches, and the instructors. These direct stakeholders

were the people who oversaw the program. They worked together to create the schedule and the

curriculum for all the students. The graduate program coordinators were important to the

success of the program because they, together with the college coaches, were the ones who

recruited the students. The GPCs were responsible for contacting school counselors and

community organizations when it was time for them to nominate eligible students. They were

also in constant contact with the instructors, campus partners, and the program directors to make

sure that all the materials and facilities were prepared for the duration of the program. All direct

stakeholders were in constant contact with the students; they made sure that the program ran

smoothly.

The Director of the Office of First Year Experience, the high school counselors and the

families/supporters of the students are important, because these individuals create the support

system that is needed for the students, in addition to assisting in the programming efforts for

SEaL. The SEaL program in particular is a program that is free to all students, it requires

funding from the institution, grants, and donations. It is with the help and support from the

indirect stakeholders that the program is able to run every summer. The school administrators,

counselors, and parents/supporters of the students are crucial to the success of the program,

without their trust and support; there would be no students in the program.


Review of Literature

A significant body of literature reviewed supports the notion that after school programs,

along with summer programs, can significantly aid a students social skills development.

After-school program implementation has increased drastically in the United States over the

past 15 years and there is widespread support in the education literature that such programs can

be a positive asset for children and communities (Miller, 2012, p. 36). In an assessment of the

Richardson Community Center entrepreneurial program, Miller (2012) concluded that the

programs success was highly due to its ability to foster and cultivate productive relationships.

Those relationships seem to be positively correlated with attaining social capital networks.

Lauvers (2012) article on student success and after school programs also emphasizes the

importance of students social development through expanded learning programs. Lauver (2012)

describes work done by Durlak and Weissburg (2007), who analyzed 73 after school programs

servicing students between the ages of five and 19. The authors found that after school programs

aiming to support social skills development were successful in doing so. That success was

mainly due to the programs effectively enhancing confidence among the students, promoting

positive behaviors toward peers and adults, reduced aggression, noncompliance and conduct

problems along with drug use (Lauver, 2012, p. 42).

The findings of Wright et al. (2009) supported an increase of prosocial behaviors and

bonding with peers for youth from low-income communities (p. 74). Through conducting

research centered on prosocial settings and youth, along with observing various after school

programs, Wright et al. found peer-relationships a contributing factor to a positive social

adjustment and outlook. The authors also found that as peer social support increased, prosocial

behavior and self-esteem increased as well (p. 87).


These studies insinuate that well structured and intentional after school programs support

students social development. The social development aspect is central to the mission of the

SEaL program and for that reason it will be the main focus of this evaluation plan.

Logic Model

In order for this evaluation plan to follow some sort of a sequence/order, we created a

logic model (see Appendix C). This logic model will guide our evaluation plan through its

clearly defined inputs, outputs and outcomes. Since this evaluation plan is outcomes focused, we

have identified the short, medium, and long-term social skills development outcomes we will be

assessing.

The logic model first outlines the inputs of the SEaL program that are divided into three

categories: stakeholders, Loyola University personnel and university resources. Next, the model

describes outputs, which were generated by inputs. The outputs pertain to activities and

participation. The activities section under outputs depicts the academic and social activity

involvement of the SEaL students. The activities component provides the reader with a sense of

what the program entails and how the students engaged in the program. The participation section

under outputs solely highlights the students as the main participants and identifies their

characteristics. Following the outputs, the logic model describes the programs outcomes. The

outcomes section is divided into three categories: short, medium, and long-term outcomes. The

intentionality behind the three categories is to showcase the immediacy of some outcomes versus

the more progressional development of others.

All of the highlighted elements from the short, medium, and long-term outcomes are

social skills that the SEaL program aims for its students to develop. The short-term outcomes

centered on developing professional behavior, building relationships and developing


interpersonal skills, are outcomes that the students are able to accomplish throughout the SEaL

program. The medium and long term outcomes related to sharing knowledge with peers/family,

enhancing social skills, fostering cultural competencies and cultivating networking skills, are

skills that can take a longer time to develop. Through this evaluation, we would be able to assess

whether the program is meeting its objective of developing students social skills; skills that the

SEaL program truly anticipates for its students to continue to practice and implement throughout

their lives.

The SEaL program strived to accomplish its learning goals/ objectives by incorporating

all of the elements of a logic model (inputs, activities, outputs, and outcomes). The inputs of the

SEaL program were the driving forces of the program as well as its main contributors.

Evaluation Approach

The nature of the SEaL program requires us to utilize a formal outcomes oriented approach.

Fitzpatrick, Sanders and Worthen (2003) described a formal evaluation approach as a structured

and reliable way of gathering systemic data. Additionally, the SEaL programs outcome we

choose to evaluate is the students social skills progress at the end of the program. Fitzpatrick

et.al explain that, outcome or impact studies are concerned with describing, exploring or

determining changes that occur in program recipients as a result of a program (p. 26). In the

case of the SEaL program, our outcomes evaluation approach is geared towards assessing the

development of the students social skills, which we defined earlier. Since the SEaL program is

inclusive of multiple grade levels (9-12 grade), we will invite all 135 students to participate in a

pre- and post self-administered survey.

The evaluation plan will utilize a mixed-methods approach to evaluate the students

social outcomes related to participating in the SEaL program. We will follow an explanatory


sequential design where we will first collect and analyze quantitative data in the form of pre- and

post-surveys. In the second phase of this explanatory sequential design, we will collect and

analyze the qualitative data from focus groups. Since our quantitative data will be more

dominant, the qualitative data will be utilized to support and explain the quantitative data.

Research and assessment pertaining to students social development in such programs,

according to Durlak and Weissberg (2007) has been overlooked, in terms of formal evaluation

(p. 5). Hence, the reason why we want to assess to what extent the SEaL program is meeting its

objective to further develop students social skills.

Some of the weaknesses pertaining to the outcomes approach include environmental

variables that are out of our control, which can impact the students social skills development.

Some of those variables may include social encounters that students might experience during

their commutes to and from the program, along with external social encounters that take place at

the students residence. Again, we cannot account for such experiences, because this evaluation

plan does not encompass what happens to students outside of the SEaL program. Some other

weaknesses pertain to students preconceived notions of the identities of others (e.g. gender, age,

sexual orientation, and race) and how those notions can impact their social skills development

(e.g. being respectful of others cultural backgrounds) during the program.

Quantitative Approach

Initially, students and parents will be notified of the opportunity for the student to

participate in a pre-survey through an email announcement (see Appendix D1). The students and

parents will receive a consent form titled Pre-survey Permission Form (see Appendix E1).

Within this form the students will be invited to participate in the pre-survey, as well as be


informed of the benefits linked to their participation. Additionally, parents and students will be

informed that students personal information will also be released to the evaluators of the SEaL

program. Incentives for the pre-survey will be in the form of school supplies (e.g. water bottles,

Loyola T-shirts, and Loyola pens).

Prior to students taking the post-survey, they will be asked to obtain a signed permission

form from their parents or legal guardians, allowing them to participate in the post-survey (see

Appendix E2). Students will receive an incentive for taking the post-survey as well. That

incentive will comprise of students being entered in a raffle where five students have the chance

of winning a $20 Visa gift card. Each student will be assigned a random number on two sheets

of small paper. Each paper will be stamped with an LUC signature mark for authentication.

After which, one of the papers will be placed into a large bin, from which a graduate program

coordinator will randomly select five winners. The other stamped paper will be left with the

student as a way of identifying the winners, and trying to limit fraud.

Survey instrument

As a disclaimer, prior to the implementation of this evaluation plan, we will consult with

the Institutional Review Board (IRB) since the majority of our student population is comprised

of minors. The survey instrument we created for the purposes of this evaluation project is a

modified version of the current pre- and post- SEaL summer 2014 survey (see Appendix F). The

survey content and style (i.e. questions) are adapted from the Social Skills Assessment

Adolescents, which was found on the Adapt Behavioral Services website (http://www.adapt-

fl.com/files/Social%20Skills%20Assmt-Prepare.doc.). We have tailored the questions to the

outcome we are trying to evaluate, which is the development of students social skills. The

survey will contain 29 questions pertaining to the social skills categories mentioned previously.


The survey construct map we have developed assists the reader in navigating and intricately

deconstructing each question, along with connecting the questions back to the Logic Model and

purpose behind this evaluation plan (see Appendix G). All students will be eligible to answer

every question on both pre- and post-surveys. All survey questions, with the exception of the

demographic questions and one multiple-choice question, follow a Likert scale format.

In order for us to gain a more concrete understanding of the survey and its logistics (e.g.

duration of survey) we will conduct a pilot study. The pilot study is intentional because we

would like to determine whether the participants easily understand the questions on both surveys.

We will ask three former students from the 2014 cohort to complete both pre- and post-surveys

and share any verbal feedback they might have after taking the surveys. This pilot study will

provide us with a better understanding of the time it takes to complete the surveys, and it will

also support our validity.

Survey Method & Design

The primary purpose of this quantitative survey is to evaluate whether participating in the

SEaL program helped students improve social skills (i.e. peer-to-peer interaction, public

presentation, team work, interpersonal skills, communicating with others/networking, reaching

out for help). We have chosen to implement a one group longitudinal pre-test post-test design,

because we would like to compare students level of social skills prior to entering the program

and once again after completing the program.

The pre-survey instrument (i.e. pre-test) will be administered to the students on the first

day of the program, where students will be taken to a computer lab to take the online Google

Forms survey (see Appendix H). The pre-survey will be in the form of an online self-


administered survey, which the students will receive via email. The pre-survey incentives will

be handed out to students after all students have completed their pre-survey. As mentioned

earlier in this evaluation plan, each cohort of students will have a college coach present in the

computer lab while students take the pre-survey. The college coach will be there to answer any

questions students may have regarding the technology or the pre-survey. The same principle will

apply when administering the post-survey. Each college coach will take their cohort into a

computer lab on the last day of the program where students will access their emails and take the

post-survey (see Appendix D2). Once all students have completed their post-surveys, their

names will be entered into the raffle that was mentioned earlier.

This pre-experimental design would allow us to explore if the SEaL program was

successful at meeting its objective to further enhance and develop students social skills. This

way of collecting the data will be economically efficient and will also yield a fast turnaround in

regards to the results (Creswell, 2009). Administering the survey during the last day of the

program is feasible and convenient since all students will be aware of the incentive that is to

come, if they choose to take the survey.

Survey population and sampling procedure

Our survey population will comprise of the students participating in the SEaL program of

summer 2015. Our population is a non-random stratified convenience sample. The sub-group

categories we would like to compare will be students grade level, gender, and race. Creswell

(2009) defined a stratified sample as specific characteristics of individuals are represented in the

sample (p. 148). For the intent and purposes of this evaluation plan, our survey will follow a

single-stage sampling procedure. According to Creswell (2009) a single-stage sampling

procedure is one in which the researcher has access to names in the population and can sample


the people directly (p. 148). Students confidentiality will be protected by them entering their

student ID numbers instead of their full names. As mentioned earlier, survey information will

only be used by SEaL administrators.

Statistical analysis

For the data analysis part of this evaluation plan we will incorporate a t- Test design,

which tests the statistical significance of the difference of means. The t- Test design will allow

us to better grasp and compare the relationship between our variables, which are the pre- and

post- surveys. Newcomer and Conger (2010) emphasized that t-Test utility is appropriate when

the research question at hand has a null and an alternative hypothesis along with multiple means

to test. Since our research question has a null and an alternative hypothesis, along with a pre-test

and post-test means, we will be utilizing a t-Test for our statistical analysis. The H0 (null)

hypothesis is that the [SEaL] program has no discernable impact on developing social skills

among low-income, mid-academically performing youth. The alternative, Ha hypothesis states

that there is a statistically significant relationship between participating in the [SEaL] program

and the social skills outcomes of program participants.

Our independent and dependent variables will be measured at an ordinal level. Ordinal

level measurements are appropriate for Likert scale surveys and fit with our purpose of

measuring the difference between our two means (Sauro, 2004; Newcomer & Conger, 2010).

The independent variable in this case is the matched pair. Matched pair is indicative of the

students before and after survey responses in relation to participating in SEaL. On the contrary,

the dependent variable will be the dimensions of social skills being measured (e.g. friendship

making skills, cultural competence, etc.).


Quantitative Results Presentation

The findings of this assessment will be presented in a written report utilizing a few

different methods to visually depict the information. The responses from the pre- and post-

surveys will be summarized and written in a narrative form. To bring more transparency to some

of the responses, a few students direct answer selections will be quoted. The identity of the

students will be kept confidential by not releasing any demographic information about them.

Additionally, pie charts will be employed to provide a visual representation of participants

demographic information. Bar graphs will also be employed to visually represent answers from

the Likert-scale questions. We will present the findings to the director of the SEaL program,

Karladora Chavez, with whom we will go over the results and share our thoughts on what, if

anything, needs to be done next.

Qualitative Approach

Our assessment plan also contains a qualitative component, which will further explore

students thoughts on the SEaL program, along with their experiences. The qualitative part of the

assessment will be facilitated after the students complete the post-survey. This way the students

will be predisposed to the notion of thinking about their experience during the SEaL program,

and during the qualitative component, they will have the opportunity to vocalize their thoughts

through open-ended questions. The qualitative component of this assessment plan will be in the

form of focus groups. According to Krueger and Casey (2010), a focus group is a great tool for

gathering information for formative evaluation plans. Focus group interviewing can elicit

information about what is currently working, not working, or needs improvement in a program.

They have been used to listen to participants and staff members perceptions of program

outcomes (p. 380). Since our evaluation plan is outcomes focused, employing focus group


interviewing to collect qualitative data is most appropriate. Additionally, focus group interviews

are time efficient and cost effective. The data collected from focus group interviews can later be

applied towards making improvements to a program (Krueger & Casey, 2010).

As alluded to earlier, the elements of our evaluation plan that the qualitative component

will address will be whether the SEaL program has met its goal of further developing students

social skills. The focus group platform will enable students to contribute their honest thoughts

and opinions on their experience throughout the SEaL program.

Qualitative Questions

The objective of our focus group interviews is to probe deeper into the students experiences and

explore the students thoughts on how/if the program further developed their social skills. The

focus group interviewing does not contain set parameters such as a survey questionnaire, where a

person has a limited answer selection. The open-ended questioning method contains no right and

wrong answers, but rather deeper exploration of a topic. The questions will follow a funnel type

sequencing, where the initial question will address students overall experience in the SEaL

program (Krueger & Casey, 2010). The following questions will probe at classroom

environment, projects and group activities (see Appendix I). All of these concepts are connected

to the overarching theme of the evaluation: social skills development. Additionally, several of

the questions will address the objectives of the survey questionnaire, such as communication

skills, networking skills, and friendship building skills.

Focus Group Participants

The focus groups will be comprised of students who participated in the pre-survey and

post-survey parts of the evaluation. We anticipate having a total of four focus groups. A focus

group of four freshmen students, a focus group of eight sophomores, a focus group of eight


juniors and a focus group of eight seniors. Focus group participation will be voluntary. At the

start of the SEaL program, all students will receive a consent form stating the terms and

conditions of the focus group along with the benefits of their participation to the overall program

improvements (see Appendix L). Although all students who submit a signed consent form will

have the chance to participate in our focus groups, we only want a select number of students in

order to obtain a sample of the SEaL student population. Additionally, the size of our focus

group is suitable in this case because, for some students, the topic being discussed might be

perceived as sensitive and personal (Krueger & Casey, 2010).

The focus group participants will be homogenous in nature. What homogeneity means

in this context is that the participants have something in common that relates to the topic of

conversation (Krueger & Casey, 2010, p. 382). The basis of homogeneity in the context of our

evaluation plan is that all focus group participants are SEaL students, all of whom identify as

people of color, and belonging to at risk inner city schools. At risk schools are likely to serve

a high proportion of minority and low-income students, have poor student achievement, andif

they are high schoolshave lower graduation rates (Qualified Teachers for At-Risk Schools,

2005, p.6). A strength to having a homogenous sample is the comfort that having something in

common brings to individuals, this fosters thoughtful listening (Krueger & Casey, 2010, p.

382).

Focus Group Procedure & Implementation

A focus group protocol will be used to lead the focus group discussions and ensure that

there is structure (see Appendix J). Our focus group protocol is inspired by the sample focus

group protocol by Schuh (2009) and we have adapted the protocol to fit the purposes of our

study and our sample. Being cognizant of our four grade levels (freshmen, sophomore, junior,


senior), we have decided to request four volunteers from each class. As mentioned earlier, the

number of volunteers will not be consistent due to the grade levels ranging in size. With this

being said, we anticipate to have a total of 28 focus group volunteers. According to Krueger and

Casey (2010), the size of a focus group can range from as few as four or five to as many as a

dozen people (p. 382). We want to implement a consistent focus group division and therefore,

we will try to recruit four students from each class (see Appendix K). This equates to having

four focus groups in total, where students from each of the two sophomore classes will

participate in the same focus group. The same principle will be applied to the junior and senior

level classes (see Appendix K). In the end, the size of each focus group will range anywhere

from four to eight participants. We, the authors of this evaluation plan, will serve as the

moderators of the focus group interviews. We believe this is most appropriate because, we are

not directly associated with the SEaL participants. Thus, it is our assumption that students will

not feel pressured to produce specific answers, nor will they feel afraid to speak up and share

their experiences. The power dynamic between the students and us will be very minimal. After

the students have finished their post-test survey, the instructors will notify them that it is time for

the focus group interviews. The selected 28 students, who have a signed by a legal guardian

consent form (see Appendix L), will be invited to voluntarily participate in the focus group. It

will be restated that even though they have a signed consent form, they are not obligated to

participate if they choose not to.

We have allotted 60 minutes for the focus group interviews, during which the students

will be answering six open-ended questions pertaining to our overarching research question (e.g.

social skills development). We believe the duration of the interviews will enable students from

the smallest focus group (4 students), to the largest focus group (8 students) to candidly


participate and address the six questions we have prepared. We deem 60 minutes to be an

appropriate set of time, during which all students from the largest focus group will have an equal

chance to participate more than once.

The rationale behind our questions is to address the objective of our study, as well as to

be conversational and information generating (Krueger & Casey, 2010). The moderators will

follow a question sequence, which starts with a question that opens up the conversation, after

which, the following questions target the topic at hand: social skills development. There will be

a total of six questions: four overarching questions, and two probing questions. The probing

questions purpose will be to route the conversation.

The focus group interviews will take place at Loyola Universitys Lake Shore Campus,

where available classrooms will serve as the setting for the focus groups. We will have the focus

groups audio taped so that we, the evaluators, can later transcribe the facilitations and input the

required information into coding software (Microsoft Word). There are confidentiality risks

associated with employing this practice, such as exposing the identity of the participants. We

have decided that we will be the only ones who will have access to the audio recordings.

Furthermore, while transcribing the audio recordings, whenever we come across a student name

that was exposed/ addressed, we will assign a pseudonym for that student. Once we have

finished with all of the transcriptions, all four of the audio recordings will be permanently erased.

Through implementing such tactics we are minimizing the risks associated with students

confidentiality. Parents/ legal guardians of the students will be notified of our intentions and

plans in the form of a focus group consent form, which will require their signature (see Appendix

L). Focus group consent forms will be distributed to students to take home at the start of the

program.


Moderator

As mentioned earlier, we will serve as the moderators of the focus groups in order to

make students feel comfortable enough to share their honest thoughts and experiences as they

relate to the SEaL program. Since we are the creators of this evaluation plan, we will be

prepared to keep the conversation focused on the questions at hand. We have dialogue

facilitation experience and training, which will certainly be applicable to the focus group

interviews. Additionally, we plan to prepare for the focus group interviews a few days before we

execute them, as well as follow a focus group protocol, which was mentioned previously (see

Appendix J).

We will also know when to ask follow- up questions and when to give the group some

time to reflect on a question. For every focus group, one of us will serve as the moderator while

the other as the assistant moderator, who will be responsible for taking notes and audio taping

the conversation.

Qualitative Results Presentation

The final results of the qualitative section of this evaluation will be displayed in a written report.

The report will be organized by themes such as engagement, site process, safety, cultural

experience/exposure and socializing, to name a few (Krueger & Casey, 2010) (see Appendix M

for themes/codes). The qualitative report will be comprised of a bulleted style summary list

where key findings and themes from the coding are highlighted. To further support each theme,

direct quotes from the students will be incorporated in the report as well.

Analysis Plan

The objective of a qualitative analysis is to find out what is meaningful to the purpose of

the study (Krueger & Casey, 2010, p. 396). A systematic analysis process will be implemented,

where we, as mediators, will listen for key points and reoccurring themes during the dialogue.


We will employ a descriptive coding analysis (Rogers & Goodrick, 2010). Descriptive coding

assists you in managing the volume of data by making it easier to retrieve and aggregate data

relating to a particular issue (p. 440). According to Rogers and Goodrick (2010) categories or

evaluator-generated codes can be created either before or after evaluators review all of the data.

Other decisions pertaining to the coding process can also be made either before or after data is

reviewed. For the purposes of our evaluation plan, we will develop a few initial codes that will

be tentative (see Appendix M), and as we start to apply them, we will modify them (Rogers &

Goodrick, 2010). As we are explicitly coding the transcriptions, we will employ Microsoft Word

to assist us in identifying reoccurring themes among students answers. The reasoning behind

our decision to employ Microsoft Word is to save time and accelerate the coding process.

Since all of the transcriptions will be written on Microsoft Word, we will take advantage of that

convenience and start to tag and highlight text containing some of our identified codes, as well as

new, potential codes/themes.

Limitations

Quantitative Approach

We understand that there are some limitations to keep in mind pertaining to the quantitative

approach. The number of students taking the pre-survey and essentially the post-survey should

be the same. However, we also understand that not all students who show up on the first day of

the program and take the pre-survey will be present on the final day of the program in order to

take the post-survey. The SEaL program has attempted to encourage the daily participation of

students throughout the program and in particular, by abiding by the programs Terms of

Participation (see Appendix N). Another potential limitation is the format of the survey, which

follows a Likert-scale format. In some instances the answer choices available may not be


representative of a students authentic response to a question. The way a question is worded

might also mislead a student, especially if they do not understand some of the terminology being

used.

Qualitative Approach

In addition to quantitative limitations, we also recognize potential qualitative limitations. A few

of those limitations include the environment in which the focus group interviews will be held, the

time allotted for conducting the interviews, the questions being asked and lastly, the moderators

approach to the focus group interviews. For an example, some of the focus group questions may

be better comprehended by upper classmen versus the rising 9th grade students. This could

potentially skew the results and impact validity. If students request for an explanation, this

potential limitation can be avoided by the moderator with additional clarification.

Timeline

A timeline was created with the intention to successfully implement this evaluation plan

in a timely manner (see Appendix O). It is our hope that the timeline is employed each year for

every cohort. For the purposes of this specific evaluation, we have taken into account the

incoming summer 2015 SEaL cohort and have constructed the timeline accordingly. We

intentionally backtracked our timeline to August, 2014 because, we decided that it is best to start

preparing our assessment materials for the next cohort early on. We will administer the pre-

survey and post-survey to a few volunteers from the 2014 SEaL cohort with the intention of

tailoring and adjusting the survey based on the pilot tests. We anticipate that there will be a

sufficient amount of time to prepare for the incoming 2015 SEaL cohort, which includes

obtaining an approval from the Institutional Review Board (IRB). Since the 2015 SEaL program

will begin in late June 2015 and end in mid-July, 2015 most of the work will be done during the


duration of the program (e.g. administering the pre- and post-surveys, analyze data from pre- and

post-surveys, conduct focus group interviews).

It is integral that everyone follow the designated timeline and everything be completed on

the assigned month. This way we can avoid any delays, which could potentially jeopardize the

delivery of the final report. As a disclaimer, we do understand that in some instances delays are

inevitable, and therefore, we will modify the timeline if need be.

Budget

Having a budget for a project, which in this case is in the form of an assessment, is of

high significance in order for the project to be implemented. The budget we have created for this

evaluation plan (see Appendix P) covers the projected costs we have accounted for as needed in

order to implement our evaluation. The budget can be applied to the evaluation of the 2015

SEaL program.

Since we would like to have a large number of participants for both the pre- and post-

surveys, we will allocate a larger sum of our budget towards incentives (e.g. Visa gift cards).

Furthermore, to compensate this cost, some of our evaluation-associated expenses will be either

free of charge or very cost efficient (e.g. survey administration, consent forms, college coaches,

incentives provided by SEaL program, etc.).

Next Steps

As this evaluation plan development comes to an end, there are several next steps to

consider for continuing to move forward with development of the SEaL program.

First, as the intent of this evaluation is to understand and assess the social development of the

student population in the SEaL program. Our goal is to create an understanding of the

importance of social development in summer programing in order to create innovative ways to


restructure and potentially shift the program objectives. The student experiences and

testimonials are the primary sources of information being used to move forward with the

development of the SEaL program. It is fundamental to create clear, actionable, and purposeful

plans to move forward with improvement, with the analyzed data. As the SEaL program is a

rather new program, working to implement new changes, as a result of this evaluation, will assist

in its continued growth.


References

Creswell, J.W. (2009). Research design: Qualitative, quantitative, and mixed method approaches

(3rd Edition). Thousand Oaks, CA: Sage Publications.

Durlak, J. A., & Weissberg, R. P. (2007). The Impact of After-School Programs that Promote

Personal and Social Skills. Collaborative for Academic, Social, and Emotional Learning

(NJ1).

Fitzpatrick, J., Sanders, J., & Worthen, B. (2003). Program evaluation: Alternative approaches

and practical guidelines (3rd Ed.) New York: Longman.

Jones, S. M., & Bouffard, S. M. (2012). Social and Emotional Learning in Schools From

Programs to Strategies. Social Policy Report, 26, 1-33.

Krueger, R. A., & Casey, M. A. (2010). Focus Group Interviewing. In J. S. Wholey, H. P. Hatry,

& K. E. Newcomer (Authors.), Handbook of practical program evaluation (3rd ed.) (pp.

378-403). San Francisco, CA: Jossey-Bass.

Lauver, S. (2012). Supporting Student Success Through After-school and Expanded Learning

Programs. District Administration, 48(3), 40-43.

Miller, P. M. (2012). Community-Based Education and Social Capital in an Urban After-School

Program. Education & Urban Society, 44(1), 35-60. doi:10.1177/0013124510380910

Newcomer, K. E., & Conger, D. (2010). Using Statistics in Evaluation. In J. S. Wholey, H. P.

Hatry, & K. E. Newcomer (Authors.), Handbook of practical program evaluation (3rd

ed.) (pp. 454-492). San Francisco, CA: Jossey-Bass.

Qualified Teachers for At-Risk Schools. (2005). National Partnership for Teaching in At-Risk

Schools, 2-22.

Sauro, J. (2004). Fundamentals of Statistics 1. Retrieved October 13, 2014, from Usable Stats


website: http://www.usablestats.com/about.php

Rogers, P. J., & Goodrick, D. (2010). Qualitative Data Analysis. In J. S. Wholey, H. P. Hatry, &

K. E. Newcomer (Authors.), Handbook of practical program evaluation (3rd ed.) (pp.

429-453). San Francisco, CA: Jossey-Bass.

Wright, R., John, L., Duku, E., Burgos, G., Krygsman, A., & Esposto, C. (2009). After-School

Programs as a Prosocial Setting for Bonding Between Peers. Child & Youth Services,

31(3/4), 74-91. doi:10.1080/0145935X.2009.524461


Appendix A: Manual & Student Assessment Reports 2014

Summer Enrichment at Loyola SEaL

Manual and Student Assessment Reports 2014


Table of Contents Executive Summary .....................................................................................................................................3 Mission ..........................................................................................................................................4 Program Components Pre-Program Logistics .................................................................................................................................5 Participating Schools Application Process Timelines Summary Pre-program Preparations ................................................................................................................. 10 Spreadsheets Classrooms CTA Passes Schedules Instructors Swag Family Orientation Day one Program Interactions ............................................................................................................................... 14 Classes Programs Assessment Summary .............................................................................................................................. 16 Measurement Tools Demographics Pre-Assessment Data ...............................................................................................................................18 Post-Assessment Data ...............................................................................................................................21 Assessment Results Student Testimony.....................................................................................................29 Recommendations ....................................................................................................................................31 SEaL Flyer .................................................................................................................................................. 33


Executive Summary

The Summer Scholars program, under the Office of First Year Experience, had over 130 students enroll in the program this year. The program seeks to provide prospective students a micro-college experience during the summer. Students had the opportunity to participate in a 3-week summer program. The program ran from Monday through Friday from 8:30am 5:00pm. They took classes designed for their academic level and their demographics.

Staff worked hard to create an intentional learning experience for participants, leading to the creation of three major learning outcomes. As a result of participating in the Summer Scholars program, students will:

1. Enhance academic skills and preparation for future goals 2. Understand and develop their identity as it relates to Loyola University Chicago and the

surrounding community 3. Create a long lasting bond by mentoring

Significant Outcomes Students were asked to complete a pre- and post-assessment of their experiences. Major outcomes of the post-assessment are listed below:

100% of the respondents indicated a level of confidence in their assigned academic program: Reading, Science, Math, or Writing.

100% of respondents indicated a high level confidence in the plan to attend a college or university after they completed high school.

95% of the respondents rated Activity Period a 5 or higher on the scale of 1-10. 94% of respondents indicated that they had established new relationships during their time with

Summer Scholars, whether that was their peers, the staff, and/or their faculty members. 75% of respondents indicated that they were confident in completing FAFSA.


Mission

The Summer Enrichment at Loyola (SEaL) program has joined the Summer Scholars program and will run under the Office of First Year Experience for years to come. The SEaL program exposes local high school students to a college environment and education in the diverse urban environment of Chicago at no cost. The majority of students in the program are of low income, first generation, foster youth and/or people of color. The program offers the opportunity to grow both intellectually and socially by:

Building community with peers, college student leaders, and faculty and staff; Learning the life skills and knowledge essential for college success; Immersing themselves in their academic passions both in and outside the college classroom. Enhance academic skills and preparation for future goals Understand and develop their identity as it relates to Loyola University Chicago and the

surrounding community Ultimately, the SEaL Program is committed to helping students put their best foot forward as a college applicant and future college student.

Program Components In order to achieve the program learning outcomes, the Summer Scholars program has recurring programs and curriculum aimed at achieving student development in these areas. These programs include:

Participating in a 3-week academic enrichment program Participating in College Coach-sponsored programs Academic preparation in either Mathematics, Reading, Writing, Science, and general study skills Preparation for college selection, application, and admission processes Committing to complete to program


Pre-Program Logistics

This section describes the details in planning for the SEaL program. It includes the links to the emails, flyers, and spreadsheets.

Participating Schools For the summer of 2014, the goal was to increase the number of SEaL participants radically. The objective was to have close to 160 rising 9th -10th grade students participating this summer. Due to some changes in the SEaL program, recruiting for the freshmen class was a task added a couple of months before the program began. Recruitment for rising 10th 12th grade students began in early March by contacting the schools that part took in the program in previous years. The participating high schools in 2013 are listed below:

o Summer 2013 High schools: Southland College Prep Senn High School Noble Street College Prep Kelly High School John Marshall Metro High School Gwendolyn Brooks High School Curie Metro High School Cristo Rey High School Christ the King Jesuit Prep High School

As a Graduate Program Coordinator, it is important to begin communication with school administrators, and their staff, early in the year. All the emails sent to the counselors and students have been put into a word document which can be found in the N: Drive, the document is called SEaL emails 2014: N:\FirstYearExperience\FirstYearExperience\FYE\Pre-College Summer Scholars\SEaL\SEaL emails 2014docx.docx Each school and organization was asked to submit an online nomination form with general information of their interested students who fell under our criteria. We asked that each school submit a minimum of 4 nominations per grade level- rising 9th 12th grade students. Asking for student nominations is a way gathering student contact information; however, it should not be mandatory. The counselors are encouraged to submit a nomination by a suggested date in order to begin mass emailing. The nomination form is called SEaL Nominations 2014 can be found in the Google drive: 2014 Summer Documents>SEaL> SEaL Pre-Program> SEaL Nominations 2014. After several planning meetings, the Leadership Team determined that expanding the invitation to other High Schools in the Chicagoland area would be a great way to increase the number of participants. A counselor list was created with the most updated phone numbers and emails of the counselors/principals to contact for student recruitment. Over time the list grew dramatically. As of now, the list includes elementary schools, middle schools, high schools, and community partners or organizations. The list was last updated in July 2014 with phone numbers, fax numbers and emails. The counselor contact list can be found in the Google drive: 2014 Summer Documents>SEaL> SEaL Pre-Program> SEaL Counselor Contact List. The summer of 2014 was the first year the SEaL program had a freshmen class. First Star is a program which Summer Scholars is going to adopt. Funding for the program was not where is needed to be at the time; however, we wanted to continue with this pilot program, so recruitment for the freshmen cohort began in mid-May. Something we were not conscious about was the differences in academic schedules 8th graders have in comparison to high school students. For the most part Chicago Public Schools have a


freshmen orientation that conflicted with the duration of the SEaL program. Phone calls were made in order to see which of the participating schools would be able to help recruit for the rising 9th grade class as well. We realized that the best way to recruit for the freshmen class was by contacting Charter Schools, Noble Network, and Private schools as well as the public schools. It is highly recommended to contact 8th grade teachers and principals through email and by phone. It is much more difficult for elementary and middle schools to release student information; for that reason, delivering flyers or students applications in person is a great way to build trust.

Application Process The application process this summer was changed to make things easier for all parties involved. In previous years, the counselors would receive a packet which included the applications for the students, the information packets for the parents, and flyers. This year we wanted to be more involved in the communication with the students about the program. The online application became live as of early March and the link was provided with all the recruitment materials. A paper application was created to mimic the online application, which was sent out to schools per request.

Student Recruitment During recruitment period, several school visits were scheduled per request. The schools and organizations were made aware of this option during the early stages of recruitment. School visits were normally during school hours: Monday-Friday, 8:00AM- 4:00PM. Some schools would ask that recruitment be done during college days. Occasionally community organizations requested a speaker to present during an event on weekends. During these events, the recruiter would pass out flyers, pens, and applications. Students should be contacted as soon as nominations are submitted. Flyers and mass emails should continue to be sent out to community partners, parent groups and other organizations. Some organizations to consider contacting early in the year are as follows:

Boys and Girls Club YMCA Local Churches Community Centers


College Coach Recruitment The college coaches play a huge role in this program. They work directly with the students every day for the duration of the program. They assisted in the pre-program logistics by participating in a call center in which they called all the students and reminded them about attending the program. It was very important to recruit of team that would put in the time and effort to make this summer a memorable one. The college coaches were recruited under the office of First Year Experience along with the orientation leaders. Summer Scholars became the umbrella for five different programs: Academy, Institutes, Seminars, SPELL, and SEaL. For that reason, over 30 college coaches were hired to work with all the programs. All the college coaches lived and worked at Messina Hall where they got to know one another and the students. They went through an intensive three week training which included a retreat component at LUREC.

Instructor Recruitment (Bridget Wesley can Insert more info) The SEaL program is unlike any other program. We try to provide students with a college experience yet cater to their academic needs. Instructors are carefully chosen to provide a holistic experience to the students. This summer the increase of participants allowed us to invite previous SEaL instructors back and open up the opportunity to others. Through referral based recruitment, instructors went through an interview process with the director of the Office of First Year Experience, Bridget Wesley. Along with all other summer scholars instructors, they had a one day orientation where they received their class schedules and had the opportunity to meet the college coaches who would be working with them for the three weeks.

Admissions Enrollment into the Pre-College Summer Scholars SEaL program is decided by Graduate Program Coordinator. The direct involvement with the admissions processes and procedures allow for an easier facilitation in the disbursement of information to students and ease registration issues. Applications were accepted on a rolling basis. The students would complete the application which had a short answer component which was the most important part. This was the only way to see the type of student who was applying. Transcripts where submitted by the counselors after the students were accepted into the program. The last day to submit applications was two weeks prior to the start of the program. Student application deadlines should be enforced in the future. Having more clear information on the website with set timelines of communication will help in having a more concrete list of participants. In theory this process is easy to work with; however, students would attend family orientation or the first day without having ever applied. College coaches had laptops and tablets on hand for those students to apply upon arrival. We allowed students to join the program because students would drop out of the program, allowing a seat for others. This became somewhat of an issue with the junior class. Ideally we would want no more than 20 students per classroom, but in the end the two junior cohorts had over 25 students.

Counselor Contact Timeline Counselor communication flow should be very natural. Counselors will build a sense of trust in the GPC and in the program. They will begin to feel comfortable contacting you at any point in time with all sorts of questions. It is important to send out recurring emails but also leave room for them to feel a sense of


connection. Although for summer 2014 there was a good communication flow, in the future it would be a good idea to update the instructors on who completed the program and how their students did. The following schedule of communication was used for summer 2014 for the most important information:

Nomination instructions (1-2 Days after nomination) Early March

Next Steps (1-2 Days after nomination letter) Mid March

Nomination Deadline Reminder (Once a week-Final day to apply) Early April

Admitted student list (1 week after nominations deadline) Mid April

Transcript request (included with admitted list)

Thank you for supporting (1 week prior to start of program)- soon after all information has been submitted

Student Contact Timeline Student contact should include set deadlines and dates when information will be sent. For example, student communication happened sporadically with some students receiving information only one week or a few days before orientation. The following schedule of communication was used for summer 2014:

Nomination Letter (1-2 Days after nomination) Mid March

Next Steps (1-2 Days after nomination letter) Mid March

Application Deadline Reminder (Once a week-Final day to apply) Throughout April

Admissions Letter (1-2 Days after admittance)

Forms and Program Benefits (included with admission letter)

Student Welcome/Orientation Virtual Packet (One Month Prior to Orientation) o Orientation Information o Campus Maps o Activities/Programs Info o Dress Code o Class Syllabus/Book Purchase/Supplies

Call Center ( 1 week after Family Orientation)

First Day of the Program Reminder (1 week prior to start of program) Follow-up and student confirmation should occur as deadlines are missed. Students should be informed that they will be dropped from the program if they do not confirm their attendance one week prior to the start of the session.


Summary Overall, administration should move to a more clear process of communication and outreach to students. In the future working closely with instructors, schools counselors, school administrators and community partners to develop clear procedures which will ensure that students are processed in a timely and effective manner should be initiated. Presenting the mission of the program and the learning outcomes while recruiting, to those involved, can allow for increase in student and parent interest. Below is a table with clear learning outcomes that students present after completion of the program.

Academic Identity Social Identity

Understand classroom dynamics in a university setting

Navigate between their personal values and those of the community

Develop skills and confidence to pursue future academic goals

Develop public speaking skills

Develop educational interests and pursuits

Become independent thinkers and responsible

Practice critical thinking skills Establish relationships with faculty, staff, and peers

Increase their self-efficacy in utilizing resources Understand the concept of shared space


Pre-Program Preparations

This section describes the details in planning for the SEaL program. It includes examples of emails, timelines, flyers, and spreadsheets.

Spreadsheets When working with a large group of people, keeping things organized can become a hard task. For the SEaL program, several spreadsheets were created in order to keep things organized. A list of completed student applications was created along with on for those who were nominated but never applied. These lists were used when we needed to send mass emails. They allowed us to send the proper emails to those students and to know what type of emails needed to be sent out. Out of all the spreadsheets created, the most important one was the complete student roster which included check lists of the forms the students had turned in. This list was used to create the individual cohort rosters and it was also used to give to Damen Dining for the lunch period. This list can be found in the N: Drive or the Google drive: N:\FirstYearExperience\FirstYearExperience\FYE\Pre-College Summer Scholars\SEaL\2014\SEaL- important files\SEaL Pre- Program\Seal Final Roster 2014.xlsx

Gathering materials from students Soon after admitted, they are sent a list of materials that need to be turned in prior to the program. This year we asked the students to request from their instructors to submit their most recent standardize test scores and transcripts. These pieces of papers were not utilized as it was intended this year. Ideally the instructors would be given this information ahead of time to see where their groups of students are in terms of their academics. The sheet that the students were responsible for submitting prior to the start of the program was the terms of participation form. This paper was a check list of all the things they were agreeing to; it also included a section where a parents/supporter gave permission for the students to participate in this program. All other materials or forms that were important were given during the family orientation and during the first day of the program. This included the emergency contact card, the parent agreement form and the photo release form.

Classrooms Preparing for the program also includes preparing for the instructors. When reserving classrooms or auditoriums, it is good to know ahead of time the preferences of the instructors. Some instructors preferred lecture style classrooms others would request a science lab. Whatever the case maybe, campus reservations needs a couple of months in advance if large or popular venues need to be reserved; places like computer labs, smart classrooms, the east quad, Damen mpr, or auditoriums. Reservations can be made through 25live or directly with the department- if they do not show up in the reservation website.


CTA passes The SEaL program has always provided public transportation passes to its students. One issue which was come across this year was ordering one week passes. The city of Chicago has changed it form of payment for transportation. CTA is now using the Ventra pass system. They do not sell seven day passes at retail stores nor at the train stations. Ordering in bulk directly from the company was difficult. Loyola needs 7-10 business days to create a check, soon after the check is mailed; it takes over a month for the check to be processed and 7-10 business days to receive them. Overall, this process took too long and we had to buy over 100 reloadable passes all throughout the city. Most retailers only sell a couple at a time so figuring out who has the most and who would sell most of their stock was a tough task.

Schedules The goal of this summer was to enrich the students with knowledge and experience. When creating their daily schedules, we kept in mind what we wanted them to take away from this summer. The students took classes in different classroom and different buildings. This allowed for them to explore the campus and to see what it feels like to walk around a college campus. Below is a sample of the daily schedule for two of our cohorts:


Instructors During the preparations for the program, the instructors were asked to submit a syllabus. This was to gain a better understanding on what the classes would consist of. At this time we asked them to create a supply list. However, because the SEaL program does not have set funds/budget, this supply list was more of a wish list. We would only get what we could and what was essential; for example, ACT books, novels, pencils, paper, markers, etc. The instructors were also asked to send in any printing materials that they would need. They were informed not to purchase anything on their own because the reimbursement process was difficult. Emails with program updates were sent a couple weeks prior to the start of the program. Instructors were asked to complete the mandated reporter exam online and to submit their New Hire Paper work soon after the instructor orientation. Aside for that, most of these emails contained information about class rosters and important dates.

Identification Badges The increase in students admits made us realize that the students safety was a priority. We decided to make Summer Scholars Identification Badges. The IDs were created after the family orientation because we had a better idea of who would be attending the program. The IDs consisted of the students first and last name, grade, and the Pre-College Summer Scholars logo. We handed these out during the first day check-in. The students had to wear the badge with the Loyola lanyards provided on a daily basis.

Ordering Swag Ordering ahead of time can be difficult if we do not want to purchase a surplus of materials. As stable program, we are able to make a rough estimate on how many students will apply for the summer scholars program year after year. This year we made a large order for the entire program. We ordered t-shirts in a variety of sizes. The t-shirts were designed by one of the graduate program coordinators and scheduled to arrive two weeks prior to the beginning of the summer. Lunch wristbands were ordered for the SEaL students in order to make lunch a Damen dining an easier process. More lanyards were not ordered because a large order was made from the previous years. New ramblers, pad folios, and water bottles were ordered for this year with the new logo, enough for all five programs for this year and the next.

Family Orientation Ordering ahead of time can be difficult if we do not want to purchase a surplus of materials. As stable program, we are able to make a rough estimate on how many students will apply for the summer scholars program year after year. This year we made a large order for the entire program. We ordered t-shirts in a variety of sizes. The t-shirts were designed by one of the graduate program coordinators and scheduled to arrive two weeks prior to the beginning of the summer. Lunch wristbands were ordered for the SEaL students in order to make lunch a Damen dining an easier process. More lanyards were not ordered because a large order was made from the previous years. New ramblers, pad folios, and water bottles were ordered for this year with the new logo, enough for all five programs for this year and the next.


Day one The first day of the SEaL program was designed to get the students familiar with their peers, their instructors, their college coaches and the campus. The students were explained in detail what the mission of the program was. They participated in various ice-breakers and team building activities which were led by the college coaches. They met with their instructors for 45min, took the post assessment and took a tour of the campus. On this day they were also give their swag, their IDs, and their Ventra passes. Below is the letter that was sent out electronically to all the students who had applied. N:\FirstYearExperience\FirstYearExperience\FYE\Pre-College Summer Scholars\SEaL\2014\Pre-program\SEaL Orientation.doc


Program Interactions

This section describes the program interactions amongst the students and staff and how they achieved the various intended learning outcomes.

Classes Each SEaL student took two academic courses and one college prep course. These courses were created for the SEaL Summer Scholars students only. The academic and college prep courses were taught by current high school teachers or college professors. The students had two 90 min academic periods, one in the morning and one in the afternoon and one 90 min college prep period. The instructors taught their classes during their scheduled periods while the college coaches were there to assist in any way. With classroom liaison hours to complete, the college coaches spent a lot of time with their SEaL cohorts. Throughout the three weeks, the students attended LifeSkills workshops two times a week and Reflection period three times a week. They had an Activity period after lunch every day where they had the chance to do get some exercise.


College Coach Programs Throughout the entire program, the College Coaches prepared various evening programs which were offered to the SEaL students. The programs were held at the first floor lounge of Messina Hall. The programs were initially proposed to the Graduate Program Coordinators and executed by the College Coaches. The majority of the programs were designed to be fun, interactive, engaging and educational; the topics varied day to day. The students were given a schedule off all the programs during their first day of classes. They were permitted to stay at the Hall after scheduled program hours if they turned in a parent agreement form. The students were asked to head home by 8:00PM because of Chicago Curfew hours and their safety. If students were going to get picked up, they would make their own arrangements and notify a college coach. For the safety of all the residential students and to respect their space; SEaL students were not allowed on the residential floors unless the evening program required it. A sample of the calendar and the parent agreement form can be found below:


Assessment Summary

Measurement Tools In order to measure student growth and achievement of the program outcomes, the following measurement tools were incorporated into the Summer Scholars program. Pre-Assessment Survey Prior to participation in the program, students took a survey consisting of four questions, two of which were general and two of which were grade-specific. The grade-specific questions were regarding their skill level in the various areas that they would be studying. Post-Assessment Survey On the final day of the program, the students took another survey that included questions about their activity periods, college coaches, Reflection/LifeSkills, and classes. These questions were aimed at assessing how prepared the students felt to enter college after the program was over. The survey included multiple choice and short answer responses. They were also asked various questions regarding general program feedback.

Student Demographics The racial identities of the students were as follows: 28 African American, 3 American Indian or Alaska Native, 6 Asian/Pacific Islander, 53 Latino/Hispanic, 2 White/Caucasian, and 7 identified under other. The students were all commuters and are from various schools in Chicago.


School Demographics A chart of high schools they attend are listed below. Of the 135 participants, there were 19 freshmen, 25 sophomores, 49 juniors, and 40 seniors. These numbers do not accurately reflect the demographics of all the students, since not all of them completed the post assessment.

High School (65 respondents) Percentage High School Percentage

Curie- 4 6% Oak Lawn- 1 1.5%

UNO Garcia- 2 3% Kelly- 4 6%

Noble College Prep- 6 9% Lincoln Park- 1 1.5%

Ace Technical Charter- 1 1.5% Westinghouse- 1 1.5%

Southland College Prep- 2 3% Northside College Prep- 1 1.5%

Gwendolyn Brooks- 7 10.8% Amundsen- 1 1.5%

Percy L Julian- 1 1.5% Evanston Township- 1 1.5%

Senn- 5 7.7% Young Womens Leadership 1.5%

Carl Schurz- 2 3% Oak Park- 1 1.5%

Cristo Rey- 4 6% Northtown Academy- 3 4.6%

Illiana Christian- 1 1.5% Roosevelt High School- 1 1.5%

CMSA- 5 7.7% Rauner College Prep- 1 1.5%

Von Steuben- 2 3% Riverside Brookfield- 1 1.5%

Lane Tech- 1 1.5% Hinsdale Central- 1 1.5%

Whitney Young- 1 1.5% Morton East- 1 1.5%

Providence St. Mel- 1 1.5%


Pre-Assessment Data

On the orientation day (July 24) of the program, students took a 4-item questionnaire. The assessment sought to determine where students were at developmentally and in their learning prior to participating in the program. 124 students completed the questionnaire. Of the 124 students, there were 40 rising seniors, 40 rising juniors, 27 rising sophomores, and 17 rising freshmen. Around 20% of the students who responded were returners from prior years.

Skills Freshmen Almost three-fifths of freshmen felt that their reading comprehension skills were good to excellent; whereas around two-fifths felt that their skills were ok/average. Only one student felt that their ability in this area was below average. Around 45% of the students felt that their writing composition skills were good to excellent. Another half of the students felt that their skills were ok/average. Only one student thought that they were below average in this area.

Sophomores Almost three-fifths of the sophomores felt that their reading comprehension skills were good to excellent; whereas around two-fifths felt that their skills were ok/average. Only one student felt that


their ability in this area was below average. Results for the science lab class were similar, except there were no students who felt that their skills were below average in this area.

Juniors Around a quarter of the students felt that their logic and problem solving skills were good/above average. A majority of students (71%) were ok/average, and another 5% felt like their abilities in this area were below average. The results for the Math ACT section were similar.

Seniors Three-fifths of the senior class rated their research writing ability as average, and one third of the students rated their ability as good to above average. Two other students rated themselves as below average and no students rated themselves as poor in the subject of research writing. Around one-third of the students rated themselves as above average to excellent in their personal


writing abilities. Nearly half of the students felt that they were average at creative writing, and another 15% of students felt that they were below average.


Post-Assessment Data On the final day of the program, students were given a post-assessment to fill out. The assessment asked questions regarding their classes, activity periods, and LifeSkills/Reflection periods. Of the 135 students who were in the program, 92 students took the post-assessment. 34 students were rising seniors, 21 were rising juniors, 19 were rising sophomores, and 18 were rising freshmen.

Plans after High School All freshmen, juniors, and seniors plan on attending college or university after high school. Of the sophomores, over four-fifths plan to go to college/university. Additionally, one student said that they would join the military, and two had other unspecified plans. In addition to attending a college preparation class, the students were constantly exposed to the idea of attending college through the activities and reflections that were led by the college coaches.

Classes Reading Comprehension/Writing Composition - Freshmen The freshmen class focused on developing their reading skills and confidence. The students read Perks of being a Wallflower by Stephen Chbosky during their three weeks. The results for both the reading comprehension and writing comprehension classes for the freshmen were similar. Overall, most of the freshmen felt confident in their reading comprehension and writing composition skills. Only a few of them still felt a little concerned about their ability to perform well in the classes.

Science Lab/Science Reasoning Sophomores The science lab class consisted of experiments, projects, and presentations that focused mainly on environmental issues. Overall, there was less lecturing and more hands-on work that the students completed. The science reasoning class spent a lot of focus on ta

Evaluation & Assessment Final Project

Documents

Transcript of Evaluation & Assessment Final Project