EVALUATION REPORT COVER3 - MEMconsultants! ! !!!!!i!|Spy Hop Productions! Executive Summary...

EVALUATION REPORT2010-11

Spy Hop Productions

empowering youth through creative technology

i | Spy Hop Productions

Empowering Youth Through Creative Technology:

2010-2011 Evaluation Report for Spy Hop Productions

Prepared by Mary Murray M.A. and Anne Basham, M.F.A MEMconsultants

January, 2012

i | Spy Hop Productions

Executive Summary >Introduction Spy Hop Productions is a not-for-profit youth media arts and educational enrichment center located in downtown Salt Lake City. Spy Hop’s mission is to encourage free expression, self-discovery, critical and inventive thinking, and skilled participation via the big screen, the airwaves, and online. In 2010-2011, Spy Hop served 1,945 students through classes in film, audio, music and design. This evaluation focuses on the impact of core classes delivered in Spy Hop’s classrooms to 263 students.

This evaluation report details information gathered to assess the impact of these core classes and to identify lessons learned for program improvement. Data collected included end of class youth surveys, youth journal reflections, a student focus group and a survey of program alumni.

>About Program Quality

Student feedback indicates that Spy Hop delivers programs in a fashion that is consistent with organizational goals and aligns with research-based best practices in youth programming. Spy Hop core programs afford physical, psychological and emotional safety for all participants. Each class is characterized by a welcoming and supportive atmosphere that encourages youth to actively engage in skill building. Youth interaction with adults and other youth is common, fostering a sense of belonging and ample opportunities to lead, work in small groups, and partner with adults. Finally, youth are highly engaged in planning, making choices and reflecting on their progress.

4.6

4.8

4.9

4.9

1 2 3 4 5

The instructor pushed me to do my best.

The class had a welcoming atmosphere.

The instructor did a good job overall.

I would recommend Spy Hop to a friend.

Selected Ratings of Class Qualities 1=not at all and 5 = completely

ii | Spy Hop Productions

Spy Hop was a safe place; it was a place for me to grow where I felt not only that people weren't judging me, but that they were encouraging me and fostering much-needed growth. ~ program alum

I enjoyed the opportunity to control every product I was making. Whether it was a radio spot or foley, there was never anyone really holding my hand. The teachers and mentors at Spy Hop are fantastic. They would teach me up to a certain point, then allow me to do whatever I thought was best. ~program alum It's overall the best learning experience I have ever been a part of. ~ current participant

>About Student Skills Developed

This evaluation was designed to provide insight into the extent to which Spy Hop classes foster youth skills. Ample evidence indicates that students gain technical skills in digital media beyond what they could learn in school. The evaluation also found evidence that Spy Hop encourages digital age literacy, inventive thinking, effective communication, and high productivity, all important 21st century workplace skills.

Spy Hop gives me a different experience in that it offers me tools and an environment that I do not have available at my school. It al lows me to learn hands on things about music production. ~ current participant How many kids can say “I’ve done 50 recordings, sound engineered, recorded a movie?” Not very many! I can’t believe I actually did that. ~ current participant

4.4

4.5

4.5

1 2 3 4 5

… I worked as part of a team.

… I learned how to manage the process of production from beginning to end.

... I learned how to make a creative product.

Selected Ratings of 21st Century Workplace Skills In this class...

iii | Spy Hop Productions

>About Youth Attitudes Shifted This evaluation explored the ways in which Spy Hop participation shifts participant attitudes in the realms of personal development, media awareness and community and global awareness. Data reveals that the impact of Spy Hop classes is particularly profound in relation to personal awareness of talents and interests and developing a sense of what is possible for the future. Spy Hop helps students develop positive and realistic visions and goals for their education and/or careers. Spy Hop students also gain insight into the power and impact of media on their community, affording them a new critical attitude towards interpreting these messages.

Spy Hop has shown me that I have much greater potential then I had originally ever considered. ~ program participant Spy Hop has given me more confidence in my abil it ies so my future looks a l ittle brighter in my opinion. I have learned a lot during my short time here and look forward to learning more outside the walls of Spy Hop. ~ program participant I had not really thought of using games as a means of communicating real-world social problems before my experience at Spy Hop. I am now interested in pursuing this more and becoming more involved with the Games For Change movement. ~ current participant

iv | Spy Hop Productions

>About Behaviors Influenced Spy Hop’s ultimate goals center around influencing youth behaviors in a way that supports their experiences of success in work, in the community and in life overall. This evaluation shows that Spy Hop programs foster technical skills, media literacy, creativity, innovation, productivity, teamwork, communication, and other 21st century skills that educators, employers and youth development experts consider essential for success in today’s world. Program alumni attribute aspects of their school and workplace success to their Spy Hop experiences. It is particularly notable that fifty percent of surveyed alumni reported they are currently employed in the digital media field, providing tangible evidence of Spy Hop’s ability to prepare youth for the modern workplace.

Possibly the area of most impact was in planning and organizing complex projects. Fi lmmaking is a very complex process; Spy Hop taught me how to handle a large project (from props to production schedules). This in turn impacted how I approach the other areas of l ife, from academic to business. ~ program alum Spy Hop has really given me the mindset of how to complete projects, deliver them to clients, and be accountable for my product. I was given the tools that I needed to further my knowledge and I left with a job in the industry. With only a year at Spy Hop it launched me into my career, I am now an editor and love it. ~ program alum

>About Prevention This evaluation underscores Spy Hop’s role as a community-based prevention program that decreases youth risk factors and fosters protective factors. Spy Hop reduces alienation among its participants while fostering interaction and good relationships among pro-social peers around positive activities. Spy Hop instructors provide youth regular contact with supportive adults, while the classes and Open Mic program serve as a valuable community-level protective factor that provide opportunities and rewards for pro-social participation. It is reasonable to assume that by decreasing risk factors and increasing protective factors at the individual, peer and community level, Spy Hop decreases youth risk for substance abuse, violence, and other adverse outcomes.

Spy Hop has truly helped me get out of my "bubble." It's really hard for me to talk to people I don't know and I was able to work on that while here at Spy Hop. ~ current participant

v | Spy Hop Productions

Spy Hop has only given me positive involvement. I've learned to social ize a lot better with others, and just with talking to my teacher I’ve been more positive about myself in situations where I’ve often felt uncomfortable before. I love coming to Spy Hop because it keeps me busy and its way better than sitting at home doing nothing. ~ current participant

>Conclusions and Recommendations

All findings indicate that Spy Hop’s core programs:

• are extremely strong,

• are delivered consistently with best practices in youth programming,

• develop important student skills, and

• impact youth attitudes and behaviors in ways that set them up for life success.

Additions to the current array of data collection tools would strengthen future program evaluation work. Structured observations of program delivery would complement student feedback on class qualities. Evaluation efforts should expand to encompass partner programs and the very popular Open Mic program. Finally, it would be consistent with Spy Hop values and worthwhile to invite youth to take a leadership role in data collection and interpretation.

1 | Spy Hop Productions

Contents

§ Introduction, page 2 o About the Evaluation o Evaluation Design and Guiding Questions o Program Logic Model o Data Sources and Response Rates

§ Findings, page 6 o Safe Space, Community Building and Sense of Belonging o High Expectations and Youth Choice o Student Satisfaction and Instructor Quality o Engaging Technology/Media and Artistic Expression o 21st Century Skills o Educational and Workplace Preparedness o Positive Outlook on Future and Clarifying Goals o Media Literacy and Social Change

§ Conclusions and Recommendations, page 19 o Program Qualities o Skills Developed o Attitudes Shifted o Behaviors Influenced o Prevention o Recommendations


Introduction Spy Hop Productions is a not-for-profit youth media arts and educational enrichment center founded in 1999, and located in downtown Salt Lake City. Spy Hop’s mission is to encourage free expression, self-discovery, critical and inventive thinking, and skilled participation via the big screen, the airwaves, and online. The values that guide Spy Hop’s programming include the following.

• Every experience and voice has value, and everyone deserves the opportunity to explore and express their ideas with confidence.

• A comprehensive understanding of media, how it is created and the role it plays in society is crucial to civic and social engagement.

• Learning takes place when people from diverse perspectives come together in a supportive environment of risk-taking, creativity, and respect.

Spy Hop’s organizational goals are to:

• provide high quality, safe out-of-school time programming for K-12 youth;

• foster personal and artistic expression through the use of emerging digital technologies and the media arts;

• develop 21st century educational and workplace readiness skills;

• promote positive youth development and social and emotional competencies; and

• increase media literacy, personal awareness and community and global connections.

In 2010-11, Spy Hop served 1,945 students through classes in film, audio, music and design. Spy Hop students range in age from 7 to 19, with an average student age of 15. Males make up 70% of Spy Hop’s student body, while females comprise 30% of students served.

Spy Hop core classes are delivered in Spy Hop’s classrooms and detailed in the table below. In 2010-2011, 263 students enrolled in these classes, which are the focus of this evaluation. An additional 759 students participated in the Open Mic program, but this was not a focus of this evaluation.


Through collaborations with community partner organizations designed to bring Spy Hop classes to students that otherwise would not have access, an additional 923 students were served. Classes were delivered at Spy Hop’s facilities as well as partner organizations across the Salt Lake Valley in subjects including stop motion, motion graphics, podcasts, visual storytelling and claymation. These classes were also not a focus of this evaluation.

>About the Evaluation

This evaluation report details information gathered to assess the impact of programming in the 2010-2011 school year and to identify lessons learned for program improvement. The evaluation process was a collaborative effort guided by an external evaluation consultant, with Spy Hop program staff taking a meaningful role in designing a program logic model, determining evaluation questions, collecting data and interpreting results.

Core Classes, 2010-2011

Loud and Clear Audio

Youth Documentary

Arts Pitch Nic

Media Apprenticeship

Program Seasonals

− Musicology − Loud and Clear − Youth Radio − Spy Hop

Records

− Documentary 1 − Documentary 2 − Reel Stories − Write Shoot

Ride − Spy Hop On TV

− Pitch Nic Documentary

− Pitch Nic Narrative

− Audio Apprenticeship

− Design Apprenticeship

− Film Apprenticeship

− Audio 1 − Audio 2 − Design 1 − Design 2 − Film 1 − Film 2

Community Program Partners

Decker Lake Youth Center Granite Park Junior High

Horizonte Instruction & Training Center Iron County School District

Kearns Junior High Salt Lake City School District (Title 1 Elementary

Schools) Salt Lake County Division of Youth Services

Salt Lake Early Intervention Tumbleweeds Film Festival

University of Utah


>Evaluation Design and Guiding Questions

Evaluation questions were guided by Spy Hop’s Logic Model, which outlines the organization’s intentions about program qualities, participants’ experiences and the program’s impact on participant skills, attitudes and beliefs and behaviors. The specific evaluation questions were:

• To what extent is Spy Hop delivering high quality programs (defined as consistent with the logic model and research-based best practices)?

• To what extent do Spy Hop classes foster youth skills (artistic, 21st century workplace readiness and relevant developmental assets/social and emotional competencies)?

• To what extent do Spy Hop classes decrease risk factors and increase protective factors at the individual, peer and community levels?

• To what extent does Spy Hop participation shift youth attitudes and awareness (in the realms of personal development, media awareness and community and global awareness)?

• What evidence is there that Spy Hop impacts youth behaviors and their experiences of success in work, in the community and in life overall?


>Data Sources and Response Rates Both qualitative and quantitative methods were used to collect information to help answer these questions.

Youth Surveys: Youth who participated in core classes completed an online end of class survey. Surveys included a mix of Likert scale and open-ended questions about program qualities, program impact and participant satisfaction. Online surveys were completed by 93 students from 15 Spy Hop classes, reflecting a response rate of 84%.

Youth Journals: Youth who participated in intermediate and advanced core classes completed online journals at occasional intervals during their class experience. Each journal entry consisted of responses to up to six open-ended questions designed to elicit qualitative feedback about the student experience. Two hundred and two unique journal entries were completed in ten classes.

Student Focus Group: Six current Spy Hop students participated in a focus group-style discussion facilitated by the Program Director. The focus group was recorded, transcribed and analyzed by the evaluation consultant.

Alumni Study: Beginning in 2009, Spy Hop has administered an annual alumni survey (planned in collaboration with the Utah Education Policy Center). The survey is emailed to past program participants who have “aged out” of the programs. This report includes information gathered from three years of annual survey results. Eighty-two alumni responded to this survey for the first time in 2009, 2010, or 2011. Additionally, 20 of the 2009 respondents completed the survey a second time in 2011.

Class Name Surveys/ Enrollmen

t

Response Rate

Audio 1 4/5 80%

Audio 2 5/9 56%

Audio Apprenticeship 6/8 75%

Design 1 6/6 100%

Design 2 6/7 86% Design Apprenticeship

12/12 100%

Film 1 4/4 100% Film 2 4/5 80% Film Apprenticeship 11/12 92% Loud and Clear 7/7 100% Musicology 5/5 100% Reel Stories 5/5 100% Spy Hop on TV 7/8 88% Spy Hop Records 4/9 44% Write Shoot Ride 7/9 78% Total 93/111 84%


>Findings Spy Hop programs provide a safe space that fosters community building and a sense of belonging among participants. Spy Hop is committed to creating a program atmosphere that fosters the psychological and emotional safety of all participants. Research has shown that welcoming and supportive environments that afford psychological and emotional safety are fundamentally important characteristics of high quality youth programming.1 The results of this year’s evaluation suggest this is one of Spy Hop’s greatest strengths in program delivery.

On end of class surveys, students gave some of their highest ratings to questions about class atmosphere. In fact, 96% of students agreed completely or quite a bit that “the class had a welcoming atmosphere.”2

These survey findings are reinforced by qualitative data. Past and current participants consistently described the sense of community and emotionally supportive atmosphere of Spy Hop programs.

Spy Hop is a warm and welcome environment. I have never felt out of place here. Makes me feel wanted and accepted. Which is great for my self-esteem. ~ current participant

It’s a collection of freaks and geeks; it doesn’t matter where you are from and you may be seen as a completely different person in your work or school environment, but when you come here, we are all equal and respect each other’s backgrounds and perspectives. Everyone who comes in the doors of Spy Hop joins the family. ~ current participant

1 Smith, C., & Hohmann, C. (2005). Full findings from the Youth PQA validation study. Ypsilanti, MI: HighScope Educational Research Foundation. 2 End of class student surveys elicited student agreement with various statements as rated on a five point scale, with 1=Not at all and 5=Completely. Unless otherwise noted, charts summarized the mean response based on this scale.

4.5

4.8

1 2 3 4 5

This class gave me a sense of belonging to the Spy Hop

Community.

The class had a welcoming atmosphere.

Welcoming Atmosphere, Sense of Belonging


Spy Hop was a safe place; it was a place for me to grow where I felt not only that people weren't judging me, but that they were encouraging me and fostering much-needed growth. ~ program alum

These findings underscore Spy Hop’s utility as a community-based prevention program that decreases youth risk factors and fosters protective factors. Risk factors are conditions that increase the likelihood that youth will get into trouble or expose themselves to danger. Alienation is a well-researched risk factor. Protective factors, such as good relationships with pro-social peers and interactions around positive activities,3 are safeguards that promote resiliency and enhance a young person's ability to resist risks or hazards and make good decisions. Spy Hop’s programs allow students who might not otherwise “fit in” to find a place to connect with others around positive activities, decreasing their

risk for substance abuse, violence, and other adverse outcomes.

Spy Hop has truly helped me get out of my "bubble." It's really hard for me to talk to people I don't know and I was able to work on that while here at Spy Hop. ~ current participant

Spy Hop has only given me positive involvement. I've learned to social ize a lot better with others, and just with talking to my teacher I’ve been more positive about myself in situations where I’ve often felt uncomfortable before. I love coming to Spy Hop because it keeps me busy and its way better than sitting at home doing nothing. ~ current participant

My involvement with Spy Hop also helped me funnel confusion and frustration toward something creative and positive at a time in my l ife when I really need that outlet and creative space. Without Spy Hop I would not be the person I am today. ~ current participant

3 Proven Risk and Protective Factors, www.findyouthinfo.gov/research.shtml, accessed on January 8, 2012


>Spy Hop successfully balances high expectations and youth choice.

Spy Hop has prioritized holding high expectations for its students while providing youth freedom to influence class content and pace. While research into youth programming affirms that these are qualities of highly effective youth programs, “on the ground” realities reveal that it can be challenging to do both well and simultaneously.4 Nonetheless, student survey results show that Spy Hop students are challenged to meet high standards of learning and accomplishment while also having freedom of choice and opportunities to contribute and influence the class.

Qualitative data provides a rich illustration of how Spy Hop’s classes simultaneously push students to complete ambitious projects while allowing students freedom and control over their own work.

I feel I was stretched beyond my boundaries while always being encouraged to work harder. ~current participant Spy Hop creates opportunities for me to work on/take control of projects, and I have to push myself to complete them, which makes me feel proud of myself and increases my self-confidence. ~current participant At Spy Hop, I am in control of myself, and nobody else. That is the biggest thing, that in my l ife I cannot recall having ever experienced. At home, I have requirements and expectations. At school, I have rules and homework. At my last job, I was fired for being myself. The lowered expectations for me make me raise my expectations for myself, and excel far more than I would in an environment where I do not control my situation. ~current participant

4 Smith, C., & Hohmann, C. (2005). Full findings from the Youth PQA validation study. Ypsilanti, MI: HighScope Educational Research Foundation.

4.4

4.4

4.5

4.6

1 2 3 4 5

My contributions were appreciated by others in the

My opinion influenced what we did in the class.

The class challenged me to stretch my skills.

The instructor pushed me to do my best.

High Expectations and Youth Choice


I enjoyed the opportunity to control every product I was making. Whether it was a radio spot or foley, there was never anyone really holding my hand. ~program alum I felt l ike I was part of something much bigger than participating in a class at school. We were out there making movies and we were allowed total creative control of our projects and were met with all of the know-how and teaching to turn our ideas to reality. It was very empowering to create with so few limits. ~program alum

>Spy Hop programs and instructors create high levels of participant satisfaction. Student satisfaction is particularly important for a voluntary teen program, such as Spy Hop, that requires most participants to travel to the classes on their own initiative; participant satisfaction provides one indicator that Spy Hop’s programs are high quality. On end of class surveys, 85 percent of students agreed completely that, “This class was a positive experience overall.” Alumni provided similarly high ratings of program satisfaction, with nine out of ten first time respondents selecting Excellent or Very Good as the overall quality of their program experience. These ratings go up with time away from the program, with nine out of ten alumni rating the program as Excellent three or more years after completing their Spy Hop experience.

Survey, journal and focus group comments illustrate this high level of satisfaction.

I really could not be happier with the program. It is fantastic. ~ current participant It's overall the best learning experience I have ever been a part of. ~ current participant Spy Hop changed my life. I' l l always be grateful to them for that. ~ program alum

4.9

4.8

1 2 3 4 5

I would recommend Spy Hop to a friend.

This class was a positive experience overall.

Student Satisfaction


This evaluation suggests the skills and qualities of program instructors greatly contribute to student satisfaction and the overall program experience. Responses to questions related specifically to instructor quality were among the highest on student surveys. Students also utilized open-ended questions as a means for providing unsolicited positive feedback about Spy Hop’s instructors, such as those listed below.

All the teachers are really knowledgeable but they are also fun to learn from. ~ current participant One way that Spy Hop is different from school is that my teacher in Spy Hop is way cooler than my schoolteachers. She is closer to our age so she gets us better and is really nice and cool and I think she understands us. ~current participant Where do you guys find this amazing staff? I bet you have a lab that breeds amazing people to work for Spy Hop. ~current participant There is a [more] positive atmosphere than what you get from school and work. The instructors care about you and show interest in your work. When you have a question they answer in a positive and give you advice on how to improve your work better. Where in school the teacher have so many students they don't have the time to really talk to you about your work. ~current participant The teachers and mentors at Spy Hop are fantastic. They would teach me up to a certain point, then allow me to do whatever I thought was best. ~ program alum

4.9

4.9

4.8

1 2 3 4 5

The instructor did a good job overall.

The instructor listened to what I had to say.

The instructor cared about me.

Instructor Quality


The perception of social support from adults is a well-researched protective factor, providing further evidence that Spy Hop aligns with best practices in prevention.5

>Technology and digital media engage youth and encourage artistic expression.

Spy Hop asserts that its focus on emerging digital technology and media arts is important because this content appeals to youth while affording youth the opportunity for personal and artistic expression. Surveys affirm that the current array of classes attracts and engages students and that once in the classes, these youth are encouraged to use their developing technology skills to express themselves artistically.

Survey and journal comments emphasize how Spy Hop’s classes expose students to engaging digital media that they would not otherwise be able to access. Students also linked their artistic expression to media skills developed at Spy Hop.

Spy Hop gives me a different experience in that it offers me tools and an environment that I do not have available at my school. It al lows me to learn hands on things about music production. ~ current participant How many kids can say “I’ve done 50 recordings, sound engineered, recorded a movie?” Not very many! I can’t believe I actually did that. ~ current participant

5 Proven Risk and Protective Factors, www.findyouthinfo.gov/research.shtml, accessed on January 8, 2012

4.3

4.5

4.5

4.6

4.6

4.6

1 2 3 4 5

In this class, I learned how to express myself artistically.

This class made me want to learn more about the digital arts.

In this class, I learned how to make a creative product.

The class encouraged artistic expression.

The instructor supported my artistic risks.

I was excited by the kind of technology taught in the class.

Media Arts and Expression


I have been offered a grant for college that I was exposed to because of Spy Hop. I am now looking at things around me in a new way, trying to find the story behind everyday things and how I could express my opinion through fi lm. ~ current participant I value the opportunity it gave me to explore the fi lm production process as a teenager. My experiences at Spy Hop had a large part in shaping my decisions after high school and I hope in the future to join the fi lm industry. I have a sincere gratitude for

the teachers, mentors, and peers who helped guide me through what were, for me, the defining educational and artistic experiences of my high school l ife. ~ program alum Spy Hop introduced me to the medium of fi lm - that in itself is endless in its existence for me. I've met incredible people and been a part of amazing projects and moments. It opened up my abil ity to believe in my creative and passionate self - I'm eternally grateful. ~ program alum It was at Spy Hop that I discovered my identity as an artist. ~ program alum


>Spy Hop develops 21st century workplace readiness skills. Educators, employers and youth development professionals are increasingly shifting attention to defining and developing the kinds of skills that are necessary in the modern workplace. Spy Hop has focused on four skill clusters considered essential to success in the 21st century workplace:6

• Digital-age literacy, which encompasses technology literacy, cultural literacy (the ability to value and interact with diverse cultural groups) and global awareness (understanding of how groups are interconnected and relate to each other).

• Inventive thinking, which includes self-direction (the ability to work independently, develop goals and manage time), adaptability and managing complexity, creativity (the means to produce something new and original), and risk-taking (a willingness to make mistakes with the ultimate goal of enhancing personal accomplishment and growth).

• Effective communication, which involves teamwork and collaboration, personal responsibility, interpersonal skills and interactive communication using a wide range of media and technology.

• High productivity, which includes prioritizing, planning and managing for results, effective use of real-world tools, and the ability to produce relevant, high-quality products.

Student surveys reveal how Spy Hop classes foster skills pertinent to the entire range of 21st century skills.

6 NCREL and Metiri Group. (2003). “enGauge 21st century skills: Literacy in the digital age.” Napierville, IL and Los Angeles, CA: NCREL and Metiri. (p.8)

4.1

4.2

4.3

4.3

4.4

4.5

4.5

1 2 3 4 5

... I learned to monitor progress towards my own goals.

… I gained a better appreciation of diverse perspectives.

… I learned how to ask significant questions that lead to better solutions.

… I learned how to effectively communicate my opinions and ideas.

… I worked as part of a team.

… I learned how to manage the process of production from beginning to end.

... I learned how to make a creative product.

21st Century Skills In this class...


3.3 3.4 3.4 3.4 3.4 3.4 3.5 3.6 3.6 3.6

1 2 3 4

Speak in front of a group Organize my time to complete Plan ahead to meet deadlines

Lead others to complete projects Critically analyze ideas and

Plan and carry out goals Develop innovative ideas

Convey my ideas Complete complicated projects

Work as part of a team

Workplace Readiness Skills Spy Hop contributed to my ability to...

Survey and journal comments by current students illustrate the wide variety of 21st century skills that students practiced and developed in Spy Hop classes.

[Spy Hop] has positively made me communicate more clearly. ~ current participant Being a producer for a project made me nervous, because I had to be in charge of the group, but it was a really good experience. ~ current participant I learn a lot about multi-tasking and pre-planning at Spy Hop. ~ current participant I feel l ike I'm getting better at communication… I also think that I'm getting better at working with other students, and all types of musicians that I didn't get the chance to the last time I took Musicology. ~ current participant Spy Hop has definitely so far empowered me creatively in that I now have motivation to create and an outlet. I now have the abil ity to come in, check out a camera, and work on independent projects as well as assignments. ~ current participant

Alumni were also asked how their experience with Spy Hop helped them to develop certain workplace readiness skills, and used this opportunity to rate Spy Hop especially well on a variety of 21st century skills including teaming and collaboration, managing complexity, communication, and creativity.

Not at all A great deal


Alumni comments emphasize Spy Hop’s impact on developing 21st century workplace skills.

Possibly the area of most impact was in planning and organizing complex projects. Fi lmmaking is a very complex process; Spy Hop taught me how to handle a large project (from props to production schedules). This in turn impacted how I approach the other areas of l ife, from academic to business. ~ program alum Spy Hop has really given me the mindset of how to complete projects, deliver them to clients, and be accountable for my product. I was given the tools that I needed to further my knowledge and I left with a job in the industry. With only a year at Spy Hop it launched me into my career, I am now an editor and love it. ~ program alum

>Spy Hop sets students up for educational and workplace success.

3.1

3.3

3.4

3.5

3.5

3.6

3.6

1 2 3 4

Spy Hop helped me to develop a professional network.

My Spy Hop experiences helped me to be knowledgeable about industry standards

related to my course work.

My Spy Hop experiences helped prepare me for the requirements of course work.

Spy Hop programs provided me with the right types of experiences to help me succeed in postsecondary education.

Spy Hop prepared me to be knowledgeable about professional norms and standards in

the workplace.

Spy Hop provided me the right types of experiences to be successful in my current

(or most recent) job.

Spy Hop prepared me to explore career options.

Preparation for Workplace and Educational Success


Alumni reflections provide valuable evidence of the ways in which their Spy Hop experience prepared them for success in the workforce and higher education. These survey findings and alumni comments are supported by the fact that fifty percent of surveyed alumni reported they are currently employed in the digital media field, providing tangible evidence of Spy Hop’s ability to prepare youth for the workplace.

Alumni comments elaborate on how Spy Hop prepares students for educational and workplace success.

What I value most is that Spy Hop taught me skil ls that normally I would not have been able to acquire until after high school. When I moved into college, I felt that I had a greater advantage in completing coursework due to my past opportunity to experiment with various types of media at Spy Hop. ~ program alum Spy Hop courses allowed me to explore and make many critical mistakes in the industry early so that I may learn to avoid those mistakes in my professional endeavors. ~ program alum


>Spy Hop promotes a positive outlook on the future and helps youth clarify their professional goals. Spy Hop aspires to foster developmental assets such as positive identity, which includes personal empowerment, self-esteem, sense of purpose and a positive view of the future. Research has linked programs that foster this developmental asset with a wide range of improvements in a youth’s life, including improved feelings of self-confidence, increased school achievement, decreased susceptibility to peer pressure and decreased depression. 7

Student survey ratings and comments reveal that Spy Hop is particularly effective at helping students set goals and develop a positive outlook for the future.

[Spy Hop] has verif ied my feelings that I do want to go into the digital media field. It isn't the easiest field to make it into, but if I work hard I can be successful. ~ current participant This has put me in a mind set of pursuing in the music business and has really helped me think about my future. ~ current participant Spy Hop has shown me that I have much greater potential then I had originally ever considered. ~ current participant Spy Hop has given me more confidence in my abil it ies so my future looks a l ittle brighter in my opinion. I have learned a lot during my short time here and look forward to learning more outside the walls of Spy Hop. ~ current participant It's made me think of what I want to study in college and what I want to be a part of for the rest of my l ife. ~ current participant

7 Scales, P.C & Leffert, N. (2004). Developmental Assets: A Synthesis of the Scientific Research on Adolescent Development. Minneapolis, MN: The Search Institute.

3.9

4.1

1 2 3 4 5

... increased my confidence.

... helped me clarify my goals for the future.

Positive Identity This class...


>Spy Hops programs foster media literacy as a tool for community and social change.

Spy Hop classes aspire to help students to become critical consumers of media messages and create their own media messages as a means of influencing their community. Student surveys ratings suggest that Spy Hop achieves these goals, although to a lesser degree than other Spy Hop goals detailed in this report (as indicated by lower average scores and qualitative responses).

Current student survey comments make the connection between student media literacy and the desire to make a difference in the community.

I am much more observant with the process when I see media, whether it is on TV or in a magazine, etc. I always try to think about how they went about to create it, and what it does or doesn't do right to make it an effective piece. ~ current participant I had not really thought of using games as a means of communicating real-world social problems before my experience at Spy Hop. I am now interested in pursuing this more and becoming more involved with the Games For Change movement. ~ current participant My involvement with Spy Hop not only improved my skil ls as a fi lmmaker but also showed me how I could use fi lm as a vehicle for social change -- it did that, and for that alone I am absolutely indebted to the Pitch-Nic Program. ~ current participant I learned the importance of reaching out to your community. While fi lming my documentary about youth’s perspective of religion I found how important connecting with people is. ~ current participant Spy Hop increased my abil ity to be social ly aware, which in turn made me realize the difference I can make in the world. ~ current participant


Interestingly, current students also point out that Spy Hop is typically more successful at fostering student awareness of their immediate community than it is at fostering global awareness or student motivation for global change.

If I think about my experience, I don’t think I learned about what sound engineers in Australia are doing; I’m not aware outside the US what is going on in my department. ~ current participant We talk about what is going on in the US, but not as much as global; we talk about what is going on in our communities. ~ current participant I would rather have community awareness; we’re all young-ish and the community is more going to impact us than the global part. ~ current participant I feel l ike that [global awareness] is beyond Spy Hop. ~ current participant

Alumni survey ratings on indicators of media literacy parallel those from current students; these ratings give positive, but relatively lower ratings to questions related to fostering student ability to change society.

However, alumni comments suggest that Spy Hop does lay a foundation for using media for social change that, in some cases, manifests years after Spy Hop participation.

Spy Hop was a huge influence in my l ife. After my documentary workshop I became really interested in becoming not only a fi lmmaker, but someone who could help tell stories that could change the world. ~ program alum Spy Hop gave me the confidence to make it in the world and make a difference.. I wouldn’t have had a chance to make a difference in this world [without Spy Hop]. ~ program alum

3.1

3.4

3.4

3.5

1 2 3 4

Understand issues and problems facing society and how I can make

Feel that I can make a difference in this world

Use media as a tool for social change

Interpret media messages with a critical eye

Media Literacy for Social Change How much did Spy Hop help you to...


Conclusions and Recommendations All findings indicate that Spy Hop’s core programs are extremely strong, are delivered consistently with best practices in youth programming, develop important student skills, and impact youth attitudes and behaviors in ways that set them up for life success.

>Program Quality This evaluation set out to determine to what extent Spy Hop delivers high quality programs. Findings reveal that Spy Hop’s programs are delivered in a fashion that is consistent with best practices. For decades, youth advocates have argued for increased availability of out-of-school time programming for children and teens. Researchers have demonstrated that regular attendance at high quality youth programs is associated with desirable outcomes including improved academic performance, social skills and interpersonal behavior and decreased risky behavior and misconduct.8 The demand for increased quantity of youth programs has recently become coupled with a complementary focus on program quality. Researchers have collected evidence that

8 Vandell, D. L., Reisner, E. R. & Pierce, K. M. (2007). Outcomes linked to high-quality afterschool programs: Longitudinal findings from the Study of Promising Afterschool Programs. Report to the Charles Stewart Mott Foundation. Accessed on 12/13/2010 from http://www.gse.uci.edu/childcare/des3.html.


effective programs share certain qualities and program evaluators have shown that desirable program outcomes are by no means guaranteed by every youth program.9

Student feedback gathered through this evaluation indicates that Spy Hop delivers programs in a fashion that is consistent with organizational goals outlined in the program logic model, and that these qualities align with what research shows to be important in high quality youth programming.10 Spy Hop core programs afford physical, psychological and emotional safety for all participants. Each class is characterized by a welcoming and supportive atmosphere that encourages youth to actively engage in skill building. Youth interaction is common, allowing youth to experience a sense of belonging and ample opportunities to lead, work in small groups, and partner with adults. Finally, youth are highly engaged in planning, making choices and reflecting on their learning.

>Skills Developed This evaluation was designed to provide insight into the extent to which Spy Hop classes foster youth skills. Specifically, Spy Hop sets forth to develop student skills using digital media for artistic expression, 21st century workplace skills, and developmental assets or social and emotional competencies associated with positive youth development. This evaluation provides ample evidence that students gain skills in each of those three areas as a result of their Spy Hop participation.

Students consistently describe gaining technical skills in digital media that exceed what they can learn in school. Alumni affirm this sentiment by describing how Spy Hop prepared them to enter higher education or the workforce.

Alumni are more likely than current students to indicate that their identity as an artist emerged at Spy Hop, although even some current students describe using their nascent skills in digital media for personal expression. It may be that developing technical mastery precedes artistic expression, and many Spy Hop students are beginners. Similarly, it may require time and hindsight to recognize

9 Westmoreland, H. & Little, P. (2006). Exploring quality standards for middle school after school programs: What we know and what we need to know. Harvard Family Research Project, Harvard Graduate School of Education. Vandell, D. L., Reisner, E. R. & Pierce, K. M. (2007). Outcomes linked to high-quality afterschool programs: Longitudinal findings from the Study of Promising Afterschool Programs. Report to the Charles Stewart Mott Foundation. Accessed on 12/13/2010 from http://www.gse.uci.edu/childcare/des3.html. 10 The Forum for Youth Investment. (2009, August). Out-of-School Time Policy Commentary #15: Raising the Bar: Quality Improvement Systems for Youth Programs. Washington, D.C.: The Forum for Youth Investment.


one’s work as artistic. It should be noted that class topics such as film, are more commonly described by students as artistic endeavors compared to classes such as audio engineering.

Increasingly, those concerned with preparing students for workplace success are focusing on 21st century skills that reflect the changing demands of the workplace. Spy Hop has focused on four skill clusters: digital literacy, inventive thinking, effective communication, and high productivity. This evaluation shows evidence that students gain skills in each of these four clusters.

Finally, as a youth development program, Spy Hop asserts that youth in its programs develop social and emotional skills and competencies, or internal developmental assets, which make a positive difference in all aspects of their life. Social and emotional competencies, such as self-awareness, self-management, social awareness and relationship skills, are shown to improve student behavior, are associated with significant improvements in attitudes toward school and prepare young people for success in adulthood.11 This evaluation shows that Spy Hop is particularly effective at supporting self-awareness, by helping students set goals and form positive and realistic visions for their future. This is important, as identity formation is a central task of adolescence. Research has shown that youth who successfully develop a positive view of their personal future are less likely to experience emotional or behavioral problems, decreased sexual risking-taking and reduced violence.12

>Attitudes Shifted

This evaluation was designed to explore to what extent Spy Hop participation shifts youth attitudes and awareness in the realms of personal development, media awareness and community and global awareness. Surveys of past and current program participants show that Spy Hop heightens awareness and impacts youth attitudes in each of these areas.

The impact of Spy Hop classes is particularly salient in shifting youth attitudes and personal awareness of their own unique interests and sense of what is possible for the future. Youth currently enrolled in classes describe shifting attitudes about their potential for a positive future in their chosen profession while alumni reflect back and credit Spy Hop for helping them discover their creative potential and professional passions.

11 Why It Matters: What is SEL? Accessed on January 8, 2012 from www.casel.org 12 Scales, P.C & Leffert, N. (2004). Developmental Assets: A Synthesis of the Scientific Research on Adolescent Development. Minneapolis, MN: The Search Institute.


Spy Hop students also gain insight into the power and impact of media messages on their community, affording them a new critical attitude towards the messages they hear from media while also allowing them to question the attitudes they have developed based on this media.

Spy Hop does not focus explicitly on teaching social change or awareness of particular community or global conditions. However, by increasing media literacy and encouraging students to express their own voice, they create a setting where motivated students develop their own social change goals and use media as a tool to achieve those goals. Some students, especially beginning students, do not explore this opportunity while focusing on developing their technical skills. Students who persist at Spy Hop over time report that their experience in the program helped them explore how to use their chosen profession to make a meaningful difference in the community, which in turn, shaped their attitudes and values.

>Behaviors Influenced Spy Hop’s ultimate goals include influencing youth behaviors in a way that supports their experiences of success in work, in the community and in life overall. This evaluation is limited in its ability to draw conclusions about the long-term impact of Spy Hop for two reasons that are quite typical of youth program evaluations: the evaluation allows for only very limited long-term follow up with its students and the outcomes Spy Hop aspires to impact are simultaneously influenced by myriad individual, family, school and community level factors. Despite these real-world limitations that make it hard to conclusively isolate Spy Hop’s long term impact, the data collected provided ample evidence that Spy Hop makes an immediate impact on youth skills, attitudes, and risk and protective factors that research shows are likely to impact the long term success of youth. Alumni feedback provides additional evidence that Spy Hop’s impact is lasting.

Spy Hop programs foster technical skills, media literacy, creativity, innovation, productivity, teamwork, communication, and other 21st century skills that educators, employers and youth development experts consider essential for success in today’s world.13 Some experts argue that few K-12 schools are able to fully prepare students for college, careers and civic education, and that many students need additional preparation in these areas to handle the increasingly higher expectations of postsecondary education, employment and civic

13 Partnership for 21st Century Skills, accessed on January 8, 2012 at www.p21.org


responsibility.14 Spy Hop plays a valuable role in preparing youth by complementing the education provided by the traditional school system, and setting them up for postsecondary success.

Alumni attribute increased levels of success in school and the workplace to their Spy Hop experiences. It is particularly notable that fifty percent of surveyed alumni reported they are currently employed in the digital media field, providing tangible evidence of Spy Hop’s ability to prepare youth for the modern workplace.

>Prevention

Spy Hop is interested in understanding the extent to which their classes decrease risk factors and increase protective factors at the individual, peer and community levels. Risk factors are conditions that increase the likelihood that youth will get into trouble or expose themselves to danger. Protective factors are safeguards that promote resiliency and enhance a young person's ability to resist risks or hazards and make good decisions. This evaluation underscores Spy Hop’s role as a community-based prevention program that decreases youth risk factors and fosters protective factors. 15

Spy Hop reduces alienation among its participants while fostering interaction and good relationships among pro-social peers around positive activities. Spy Hop instructors provide one-on-one mentoring for all of their students, which in turn provides youth with the opportunity to develop a strong relationship with a caring adult. Spy Hop make a valuable contribution to community-level protective factors by providing opportunities and rewards for pro-social participation. Although the Open Mic program was not a focus of this evaluation, it is a drop-in program open to all youth, and expands Spy Hop’s reach as a community asset well beyond those students enrolled in its other core programs.

It is reasonable to assume that by decreasing risk factors and increasing protective factors at the individual, peer and community level, Spy Hop decreases youth risk for substance abuse, violence, and other adverse outcomes.

14 Association for Career and Technical Education, National Association of State Directors of Career Technical Education Consortium and Partnership for 21st Century Skills. (2010) Up to the challenge: The role of career and technical education and 21st century skills in college and career readiness. 15 Proven Risk and Protective Factors, www.findyouthinfo.gov/research.shtml, accessed on January 8, 2012


>Recommendations This evaluation focuses on the program delivery and impact of core programs, which impacted only 14% of students served by Spy Hop. Evaluation efforts should expand to encompass more of Spy Hop’s programs. Spy Hop has begun data collection with partner programs, and should continue this effort to understand which of the positive findings detailed in this report generalize when programs are delivered in a much shorter time frame and in a different setting. Also, the very popular Open Mic program should be evaluated.

Spy Hop should reflect on its program logic model and program delivery in light of the lessons learned of this evaluation, and determine if refinements are appropriate. In particular, it is worth considering the roles of artistic expression and community and global awareness in curriculum goals. The evaluation found evidence that Spy Hop makes a difference in these areas, but findings were not as robust as other aspects of the program logic model.

This evaluation relied solely on student feedback to draw conclusions about program quality. Future evaluations should incorporate additional sources of information on program quality, such as structured observations using a tool designed for this purpose. A sampling strategy using such a tool could better uncover opportunities to develop particular instructor skills or identify ways that the program overall could bolster program delivery through targeted training and professional development.

It would be consistent with Spy Hop values and worthwhile to invite youth to take a leadership role in the program evaluation process. Youth can assist with data collection, by interviewing peers or facilitating focus group discussions. Youth can also review data and support interpretation of the findings. Spy Hop should create tools and practices that support youth to collect data, review evaluation findings, and make recommendations for program improvement.

Finally, Spy Hop should celebrate its program successes and work to insure they are sustained. This evaluation suggests that Spy Hop employs a strong staff that utilize effective program delivery strategies; since the field of youth work is characterized by high turnover, Spy Hop program managers should take efforts to retain and further develop the staff they have. Also, given the present day challenges of obtaining and sustaining funding for youth enrichment programs, Spy Hop should be acknowledged for securing sufficient resources to deliver a meaningful and impactful out of school time youth experience.

2010-11SPY HOP PRODUCTIONS | 511 W. 200 S. #100 | 801.532.7500 | SPYHOP.ORG

EVALUATION REPORT COVER3 - MEMconsultants! ! !!!!!i!|Spy Hop Productions! Executive Summary...

Documents

Transcript of EVALUATION REPORT COVER3 - MEMconsultants! ! !!!!!i!|Spy Hop Productions! Executive Summary...