Did it Work? Evaluating Youth Outcomes in Relationship ...1).pdf · Evaluating Youth Outcomes in...
Transcript of Did it Work? Evaluating Youth Outcomes in Relationship ...1).pdf · Evaluating Youth Outcomes in...
Collaborative Mentoring Webinar Series
Did it Work? Evaluating Youth Outcomes in Relationship-Based Programs
December 19th, 2013
Sarah Kremer,
Friends for Youth
Michael Garringer,
Education Northwest
Dana Gold, MP of
SWPA
Polly Roach, MP of
Minnesota
Tammy Tai,
MENTOR
Molly Brenner,
MENTOR
Meghan Ferns,
Oregon Mentors
2013 Collaborative Mentoring Webinar Series Planning Team
Good to Know…
After the webinar, all attendees receive:
Instructions for how to access PDF of presentation slides and webinar recording
Link to the Collaborative Mentoring Webinar Series webpage, where all slides, recordings, and resources are posted.
Please help us out by answering survey questions at the end of the webinar.
• All attendees muted for best sound
• Type questions and comments in the question box
• Respond to polls
• Who is with us today?
Participate in Today’s Webinar
Panelists on Today’s Webinar
Michael Garringer Resource Advisor The National
Mentoring Center at
Education Northwest
Celeste Janssen
Executive Director
Oregon Mentors
Tom Drummond
Co-Director
College Dreams
Challenges & Opportunities with Evaluating Youth Outcomes
Program Effectiveness: The Pathway to Improved Youth Outcomes
How well you run
your program
How well your
volunteers mentor
the youth
Quality of match
relation-ships
Mentee’s attitudes on certain
topics
Youth behavior
(outcomes you wish
to achieve)
Jean Rhodes’ Model of Youth Mentoring
What Makes Evaluating Relationship-Based Programs Unique?
• Multi-modal benefits create lots of opportunities for measurement and challenges in knowing your impacts
• The relationship can be tricky to measure
• The activities mentoring programs provide and the outcomes they measure aren’t necessarily aligned with policymaker goals
Three Concepts That Can Help Answer “Did It Work?”
• Theory of change
• Deeper understanding of relationships
• A focus on intermediate outcomes and measures
Theory of Change
• Similar to a logic model, but more detailed and robust – Clarifies causality toward end goal
– Challenges assumptions about the intervention
• Works backwards from the ultimate outcome – Each outcome step is connected to an
intervention component or assumption
– Illustrates the causal pathway to long-term outcomes
TOC Example
Source: Aspen Institute
TOC Example
Source: Aspen Institute
TOC Example
Source:
Aspen Institute
Example of TOC in Mentoring
• Long-term goal: HS Readiness
↓ What competencies and attitudes are needed?
↓ Activities mentors-mentees do together? Parent role?
↓ How you train your mentors to do this? Other supports?
↓ Recruiting the right mentors
↓ Bringing the right youth into the program
Resources for TOC
• Center for Theory of Change
• The Community Builder’s Approach to Theory of Change: A Practical Guide to Theory Development (Aspen Institute)
• On Logic Models Theories of Change and Evaluation
• An Introduction to Theory of Change (Harvard Family Research Project)
• Kellogg Foundation Logic Model Development Guide – Chapter 3
Understanding Mentoring Relationships
• More information about matches = Better understanding of outcomes (good and bad)
• Go beyond measures of “satisfaction” – get inside the “box” of the mentoring relationship
• Participant characteristics are a logical starting point: – Risk factors and assets
– Other supports
– Personality traits
– Personal history and past relationships
Commonly Collected Match Data
• Meeting frequency and intensity – 60% of matches meet 4 hrs. per month for one year
• Match activities (both the “what” and associated feedback) – 52% of matches engaged in 4 or more hours of service of
learning
– 60% of youth self-report an overall positive perception of the mentoring relationship
• Match duration – Average match length is 14 months
Additional Data
• Parent engagement and perceptions
• Input of teachers, other adults
• Support seeking by participants – Topics
– Frequency
– Level of staff support
– Resolution
Participant Perceptions
• Field has moved away from looking at mentors’ approaches through a “prescriptive” or “developmental” lens.
• Now emphasizing a “relational” or “instrumental” spectrum
– Pairs start at one end of this based on program model
A Framework for Understanding Interactions
• Focus – Goal-directed or relational
• Purpose – Youth (immediate, playful) or Adult (distal, progress)
• Authorship – Unilateral, reciprocal, or collaborative
• These three variables determine relationship quality and satisfaction, predictive of participant outcomes – Balance of relational and instrumental elements creates
satisfaction – Authorship (youth “voice”) is essential for receptiveness
Further Exploration of Relationships
• Internal Match Quality (relational and instrumental) – Compatibility (traits and preferences) – Competence (mentor’s approach and skills) – Satisfaction (closeness, connectedness, perceived support)
• Match structure – Purpose/goal orientation (goal setting) – Focus of activities – Authorship (one-sided or negotiated)
• External Match Quality – Program support – Parent/guardian engagement – Support network – Logistical challenges
Tools for Measuring Relationships
Youth Mentoring Survey (youth perceptions) – Harris and Nakkula
• Accurately measures match structure and quality
• Likert & short answer (grades 3-12)
• Subscales: – Fun focus
– Sharing focus
– Instrumental focus
Tools for Measuring Relationships
Match Characteristics Questionnaire (mentor perceptions) Harris and Nakkula • Also measures structure and quality • Likert and short answer • Subscales:
– Closeness – Satisfaction – Openness to Support – Academic Support Seeking – Fun, Sharing, Future Outlook, Character Development,
Education
Tools for Measuring Relationships
• YMS and MCQ can be accessed at: http://www.mentoringevaluation.com/Tools.htm
• Learn more:
– Play, Talk, Learn: Promising Practices in Youth Mentoring (New Directions for Youth Development, Summer 2010)
Tools for Measuring Relationships
Youth Survey (variation on the Youth-Mentor Relationship Questionnaire developed by P/PV)
• 19 items (youth only)
• Measures: – Youth centeredness
– Youth’s emotional engagement
– Youth dissatisfaction
• Download through the OM Toolkit or on the EdNW website
Tools for Measuring Relationships
Performance Measurement Packet: Mentoring Children of Incarcerated Parents • Two age groups: 9-12 and 13-18 • A survey for mentors • A survey for a second adult who knows the
mentee • Intermediate Outcome: Trusting mentor-mentee
relationship • End Outcome: Positive attitude toward the future
Additional Reading on Relationships
Handbook of Youth Mentoring (2nd Edition)
– Assessing Mentoring Relationships
– Mentoring Activities and Interactions
– Goals in Mentoring Relationships
– Social Networks and Mentoring
– And more…
Intermediate Outcomes
• You must make the case that your intermediate outcomes will lead to long-term changes (“needle moving”)
• Two excellent examples:
– Putting Middle Grades Students on the Graduation Path: A Policy and Practice Brief
– Out of School Time Issue Brief (United Way)
Intermediate Outcomes that Fit Mentoring
• Connectedness
• Self esteem/competence/motivation
• Future orientation and goal setting
• Peer relationships
• Adult relationships
– Support seeking
• Identity development
Where to Find Instruments
• From Soft Skills to Hard Data (Forum for Youth Investment)
• Outcome Indicators Project (Urban Institute)
• Beyond Content: Incorporating Social and Emotional Learning into the StriveTogether Framework (Strive Network) – Competencies related to academic achievement – Measures by stage on the Cradle-to-Career Continuum
• Oregon Mentors Evaluation Instrument Toolkit!
Tying It All Together
You can answer “Did it work?” by: • Showing how your program’s work feeds into
larger outcomes
• Connecting all those participant and match characteristics to outcomes – Who did the program work for (or not)? – What correlations help you improve the model? – Where can we claim some causality?
When you will need a professional’s help…
• When you want to run statistical correlations and understand relationships between many factors and outcomes.
• When you need to develop a new data collection instrument.
• When you want to establish causality in your results (experimental design, RCT or comparison group) – They can also help develop that Theory of Change!
Example from the Field: College Dreams
Tom Drummond, MSPsy
Co-Director College Dreams
www.collegedreams.org
Guiding Lights Theory of Change
Long-term goal: X ↓ What competencies and
attitudes are needed? ↓ Activities mentors-
mentees do together? Parent role?
↓ How you train your mentors to do this? Other supports?
↓ Recruiting the right mentors
↓ Bringing the right youth into the program:
Long-term goal: College Access ↓ High School Completion & Academic
Mindsets ↓ Students begin with an Academic
Motivational Interview, and meet weekly with their mentor at the YMCA or Boys & Girls Club site
↓ Academic motivational enhancement and skill training for mentors
↓ Recruit & Match volunteers with a college degree
↓ Enroll MS students with: • Academic potential (achievement test
scores of 25th to 99th percentile, no modified classes)
• At least three dropout risk factors • A Grade Point Average (GPA) between
0.00 and 2.49
Youth Outcomes in College Dreams’ Guiding Lights Mentoring Program
Grade Point Average Guiding Lights Comparison
Grade Point Average (Baseline & Current) 1.63 to 2.05 1.70 to 1.76
GPA Improvement 0.42 0.06
Percent of Students with 0.10+ GPA Gain 68% 48%
School Attendance Guiding Lights Comparison
Absence Rate (Baseline & Current) 9.2% to 6.1% 9.1% to 15.0%
Change in Absence Rate 33.6% Decrease 64.8% Increase
Improved or Perfect Attendance 72.2% 49.0%
Perfect Attendance 19.4% 10.3%
Undesirable Outcomes Guiding Lights Comparison
Any Undesirable Outcome 2.6% 14.1%
Not Attending School 0.0% 3.3%
Alternative/Remedial Programs 2.6% 10.1%
Evaluation Toolkit Domains of Interest Live Demo of Toolkit & Instruments
Evaluation Toolkit Domains of Interest Live Demo of Toolkit & Instruments
Comprehensive Surveys:
• Youth Outcome Survey
• Youth Risk Behavior Surveillance System
Connectedness:
• Hemingway Scale of Adolescent Connectedness
School and Academics:
• Education Expectations and Aspirations Scale
• Motivated Strategies for Learning Questionnaire (MSLQ)
Mentoring relationship quality:
• Match Characteristics Questionnaire (MCQ)
• Youth Mentoring Survey (YMS)
• Youth Survey: Measuring the Quality of Mentor-Youth Relationships
Personal Attitudes and Beliefs:
• Antisocial attitude scale
• Decision making skills scale
• Goal Setting Skills Scale
• Pro-social Involvement, Opportunities and Rewards
Tips for Administering Instruments
1. Plan for problems
2. Don’t hand-pick which questions to use
3. Don’t reword questions
4. Get parent permission
5. Provide a private space to administer
6. Don’t provide guidance
7. Administer the instruments in a consistent, uniform way
8. Keep the answers anonymous/ sort out any coding before administration
9. Plan for data storage and safety
Additional Resources
• Oregon Mentors Evaluation Toolkit
http://oregonmentors.org/library/evaluationtools/
• W.K. Kellogg Foundation Evaluation Handbook http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx
• W.K. Kellogg Foundation Logic Model Development Guide http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx
• Northwest Health Foundation Program Evaluation: Principles & Practices
http://nwhf.org/images/files/NWHF_Program_Eval_Handbook.pdf
Where do your Services fit within a Youth Development Continuum?
Remember…
After the webinar:
Please help us out by answering survey questions at the end of the webinar.
Everyone will get an email with information on how to
download the slides, recording, and resources on the CMWS webpage on the MENTOR website:
http://www.mentoring.org/program_resources/training_opportunities/collaborative_mentoring_webinar_series/
Access CMWS Information All Collaborative Mentoring Webinar Series information for upcoming and past
webinars is on the new CMWS webpage on the MENTOR website!
Receive Monthly CMWS News
Sign up for the newsletter!
• Information about
upcoming webinars
• Evaluation results from the
previous webinar
• Links to resources and
recordings from past
webinars
Topic:
Briefing: New Mentoring Report from MENTOR – Insights and Feedback
Registration will open next week on the MENTOR website.
Sarah Kremer,
Friends for Youth Michael Garringer,
Education Northwest
Meghan Ferns,
Oregon Mentors
Dana Gold, MP of
SWPA
Polly Roach, MP of
Minnesota
Tammy Tai,
MENTOR Molly Brenner,
MENTOR
Next Webinar: January 17, 2014 1:00-2:15PM EST/
10:00-11:15AM PST
2013 Collaborative Mentoring Webinar Series Thank you for participating today!