Using the DLA 20 to measure functional outcomes in a child and
READY - A Youth Development Outcomes Measure
description
Transcript of READY - A Youth Development Outcomes Measure
READY - A Youth
Development
Outcomes Measure
U N I V E R S I T Y O F
ROCHESTER M E D I C A L C E N T E R
Project Team
United Way of Greater Rochester– Kathy Lewis– Elizabeth Ramsay
Rochester-Monroe County Youth Bureau– Chris Dandino
Univ. of Rochester Medical Center, Div. of Adolescent Medicine– Jonathan D. Klein, MD, MPH – Melissa Matos Auerbach, MA– Shannon M. Smith– Sheryl Ryan, MD– Cheryl Kodjo, MD, MPH– Premini Sabaratnam, MPH
Representatives from youth serving agencies in Rochester
Participating Agencies included:
Baden Street Settlement Big Brothers, Big Sisters of Greater Rochester Boy Scouts of America, Otetiana Council Boys & Girls Club of Rocehster Center for Youth Services Charles Settlement Community Place of Greater Rochester Girl Scouts of Genesee Valley Hochstein Music School Metro Council for Teen Potential Southwest Area Neighborhood Association (SWAN) Urban League of Rochester YMCA of Greater Rochester
Youth Development
Philosophy or approach - a set of principles emphasizing active support of the growing capacity of young people by individuals, organizations, and institutions
Characterized by a positive, strength building orientation
Occurs at home, in school, among peer groups, and in community-based programs
Has gained importance nationally, in states, and in local communities
Community Outcomes
Local funders are increasingly concerned with demonstrating effective progress toward outcomes
Existing measures of youth development are lengthy and complex
Youth Development Outcomes Measurement Project
GOAL:– To develop an evaluation tool for YD
programs that met the following criteria:
Easy to Use Easy to AdministerApplicable to a Variety of Youth Development
ProgramsUseful for Assessment of Impact of Program on
Youth Development of Participants
Rochester, New York
City in Monroe County, Western New York
Population 219,773 52% non-white 37% ages 19 or less Person under age 18 in 34%
of households 32% of families with children
under 18 below poverty level
Source: US Census, 2000
Youth-Serving Agencies
Serve the Youth of Rochester through: case management counseling homework assistance sports programs life skills building leadership programs music lessons provision of safe, open recreational spaces
Three Phase Project
Phase I Instrument development via a consensus process
Phase II Piloting to test validity and reliability of instrument
Phase III Field tests and dissemination
Phase I: Instrument Development - Dec. 2000 - May 2001
Meetings with representatives from youth-serving agencies and funders
Identification of core and optional outcome measures and questionnaire items
Establishment of face validity of core measures and measurement strategies
Identification of Core Outcomes
Three meetings resulted in list of 54 indicators and 10 outcomes
peer and adult relationships constructive use of leisure time basic social skills community service health maintenance decision making process responsibility understanding boundaries/rules positive identity independent/daily skills
Narrowing List of Outcomes
Agency representatives were asked:
What impact does your program have?
What would you like to learn to improve the quality of your program?
Programs Wanted to Know
Effectiveness of staff Effectiveness of services they provide Impact on youth and their families Impact of youth involvement in more than one
program Youth Development philosophy of staff Gaps and what programs can do about them
Consensus Process
Programs used nominal iterative process to identify consensus priority areas for youth that they could impact
First Round: 18 constructs
Second Round: 7 constructs– Top 2 retained
Third Round: 4 constructs– Top 2 retained
Outcomes for Operationalization
Basic Social Skills
Caring Adult Relationships
Decision Making Process
Constructive Use of Leisure Time
Candidate Questions
Questions adapted from instruments by Add Health Boys and Girls Club of America Girl Scouts of America Metro Council for Teen Potential Worcester Youth Development Initiative YMCA
Phase II: Piloting the Draft Instrument - May 2001 - March 2002
Piloting in two phases:
A. Cognitive interviews to test validity of items
B. Field test of internal consistency of items and feasibility
A. Cognitive Interviews
48 urban and suburban adolescents aged 10 to 17 Mean completion time: 11 minutes 70% had no suggestions 67% reported survey was “easy” to complete 81% understood everything in the survey 98% did not mind answering the survey Items re-worded to increase readability (now at 4th grade
level) and to simplify concepts
B. Field Test
389 urban and suburban adolescents Ages 10 to 19 Large drop-in programs and smaller, structured
programs Findings:
– Feasible for program staff to administer– Large groups required more staff time– Easy for older adolescents– Some issues remain for younger adolescents
Field Test Results
Youth more attached to programs did better on measures
Four constructs have several good factors for program use in evaluation
Instrument consists of six factors, corresponding to three outcomes
Internal reliability scores (s from .5782 to .8557)
Factor Analysis: Core Outcomes
Self Control
Empathy Basic Social Skills
Communication
Staff Relationships Caring Adult Relationships
Program Effect
Decision Making Decision Making
Constructive Use of Leisure Time
Factor Analysis - all participants OUTCOME
CONSTRUCTStaff
RelationshipsProgram
EffectivenessSelf Control Empathy Communication
ITEMS 1 2 3 4 5 6Usually feel connected 0.094 0.657 0.111 0.121 -0.174 0.163Listen to w hat others say 0.156 0.104 0.176 0.735 0.035 0.041Think about how others see things 0.038 0.000 0.074 0.659 0.277 0.195Ask others questions to get ideas 0.070 0.164 0.167 0.354 0.576 0.031Stay out of f ights 0.073 -0.017 0.830 0.095 0.137 -0.043Keep anger under control 0.120 0.008 0.647 0.359 -0.016 0.090Stay out of trouble 0.064 0.034 0.715 0.015 0.145 0.305Ask for help if needed 0.143 0.044 0.107 0.012 0.668 0.182Program helped make/keep friends 0.214 0.647 -0.077 0.169 0.134 0.026Program helped make better decisions 0.135 0.540 0.045 -0.003 0.398 0.149Since coming to prog., relations w / family adults better 0.200 0.559 -0.008 -0.128 0.173 -0.034Talk about thoughts and feelings 0.131 0.225 0.041 0.318 0.502 0.045Would talk to program staff about problem 0.567 0.280 -0.025 0.048 0.175 0.067Program staff expect me to try my best 0.724 0.131 0.169 0.012 -0.049 0.006Program staff care about w hat happens to me 0.742 0.197 0.073 0.023 0.108 0.094Program helped identify caring adults 0.616 0.372 -0.060 0.109 0.210 -0.020Program staff listen to me 0.681 0.137 0.069 0.084 0.074 0.189Program staff act like young people are important 0.761 0.033 0.102 0.171 0.009 0.040Young people can make difference at program 0.724 -0.112 -0.014 0.152 0.129 0.094Think about how decision w ill affect my future 0.099 0.120 0.128 0.320 -0.121 0.678
Think about how decision w ill affect others 0.054 0.121 0.240 0.266 0.128 0.583
Listen to others, then make ow n decision 0.219 -0.026 0.095 0.529 0.104 0.279Think about how made decision afterw ards 0.113 0.100 -0.019 -0.014 0.241 0.722
Think about consequences of decision afterw ards 0.194 -0.323 0.065 0.077 0.402 0.467
ROTATED COMPONENT MATRIX: ALL CASES (n=389)
Caring Adult Relationships Social SkillsDecision Making
Factor Analysis - 13-19 year olds OUTCOME
CONSTRUCTStaff
RelationshipsProgram
EffectivenessSelf Control &
EmpathyCommunication
ITEMS 1 2 3 4 5
Usually feel connected 0.076 0.596 0.352 -0.069 0.042Listen to w hat others say 0.134 0.101 0.527 0.496 -0.139Think about how others see things 0.026 -0.035 0.292 0.631 0.155Ask others questions to get ideas 0.017 0.203 0.085 0.611 0.184Stay out of f ights 0.086 -0.051 0.730 0.064 0.158Keep anger under control 0.185 -0.032 0.659 0.142 0.117Stay out of trouble -0.016 0.080 0.566 0.119 0.371Ask for help if needed 0.135 0.030 0.045 0.678 0.226Program helped make/keep friends 0.240 0.634 0.008 0.268 -0.142Program helped make better decisions 0.122 0.626 -0.003 0.239 0.128Since coming to prog., relations w / family adults better 0.149 0.650 -0.121 -0.028 0.072Talk about thoughts and feelings 0.249 0.291 0.007 0.531 0.109Would talk to program staff about problem 0.473 0.364 -0.057 0.115 0.151Program staff expect me to try my best 0.713 0.132 0.251 0.014 0.001Program staff care about w hat happens to me 0.744 0.158 0.106 0.060 0.183Program helped identify caring adults 0.681 0.386 -0.057 0.080 0.020Program staff listen to me 0.689 0.185 0.101 0.143 0.197Program staff act like young people are important 0.763 0.043 0.180 -0.011 0.045Young people can make difference at program 0.716 -0.078 -0.037 0.222 0.118Think about how decision w ill affect my future 0.180 0.053 0.399 0.000 0.588Think about how decision w ill affect others 0.035 0.160 0.300 0.256 0.613Listen to others, then make ow n decision 0.192 0.000 0.171 0.181 0.518Think about how made decision afterw ards 0.113 0.137 -0.004 0.088 0.750Think about consequences of decision afterw ards 0.114 -0.226 -0.025 0.399 0.572
ROTATED COMPONENT MATRIX: AGES 13+ (n=248)
Caring Adult Relationships Social SkillsDecision Making
Factor Analysis - Reliability
FACTORStaff
RelationshipsProgram Effectiveness Self Control Empathy Communication Decision Making
0.856 0.609 0.659 0.584 0.578 0.640
ALPHA COEFFICIENTS FOR FACTOR COMPONENTS: ALL CASES (n=389)
FACTORStaff
RelationshipsProgram Effectiveness
Self Control & Empathy
Communication Decision Making
0.845 0.614 0.648 0.660 0.707
ALPHA COEFFICIENTS FOR FACTOR COMPONENTS: AGES 13+ (n=248)
Phase II: Con’t
Slight revision of individual items and rearrangement of questions leading to final instrument:
– Pencil and paper survey
– 40 questions addressing four core outcomes, program participation, connectedness to program, and sociodemographic information
– Requires between 10 and 15 minutes to complete
Development of training modules, scoring templates, and score report generating software
Phase III: Dissemination in Rochester - May 2002 - May 2003
Summer and Fall 2002 - 11 youth serving agencies in the Rochester area were trained to use the instrument and the report generating software
TA provided to agencies to develop appropriate sampling plans
During the program year of 2002-2003, over 1,000 youth participating in YD programs in the Rochester area completed surveys
Current Steps: May 2003 - present
Software and score reports revised based on qualitative feedback from Year 1 implementation
Continued training and TA to current users Dissemination to various other youth
development programs through ACT for Youth Center of Excellence
Validation studies
Additional sites
Alaska - Residential School System served as a beta test site in school year ‘02-’03
Erie County, NY - over 75 programs including youth bureaus and youth boards, and UW and Department of Youth Services funded programs
Hawaii - Children’s Alliance of Hawaii Oswego County, NY - Oswego City-County Youth Bureau funded
programs Salamanca, NY - 21st CCLC program Syracuse, NY - Catholic Charities of Onondaga County sites Queens, NY - Queens Child Guidance Center Beacon & OST
sites
Use of Data by Programs
Internal quality improvement. Examples include:– Reviewing and discussing score reports with staff and with Boards– Comparing program scores within one agency to identify opportunities
for improvement– Reviewing curricula and current programming strategies– Discussing program strategies with other similar programs– Identifying training and technical assistance needs
Reporting to funders Proposal writing Sharing data back allowing the creation of an aggregate
community level score report using de-identified data
Training and Technical Assistance
Training, TA, and use of the READY Toolkit are available to interested users
Fees are based on the number of users and the level of training and TA required
Options include:– Training and TA provided directly to end users– One time training provided for end users, and
continuous training and TA provided to a lead agency which then agrees to provide first line TA to end users
READY Toolkit
READY Toolkit includes a CD which contains:– A Personalizable Instrument Template– READY Analysis Program– Toolkit Instructions Manual– User’s Agreement
Instrument template may be personalized to contain program names and staff titles
READY Analysis program allows community programs to enter their own survey data, and generate a score report
Score report contains summary measures for core YD outcomes and frequencies for all survey items
SAMPLE Score Report
Pg 1 of 17
SAMPLE
Score Report Pg 2 of 17
Publications
Klein JD, Sabaratnam P, Matos Auerbach M, Smith SM, Kodjo C,
Lewis K, Ryan S, Dandino C. Development and factor structure of a
brief instrument to assess the impact of community programs on
positive youth development: The Rochester Evaluation of Asset
Development for Youth (READY) tool. Journal of Adolescent Health 2006;
39: 252-260. Sabaratnam P, Klein JD. Measuring youth development outcomes for
community program evaluation and quality improvement: Findings from
dissemination of the Rochester Evaluation of Asset Development for
Youth (READY) Tool. Journal of Public Health Management and
Practice 2006; 6(suppl): S88-S94.
For more information about the READY tool, please contact:
Premini Sabaratnam, MPHSr. Health Project Coordinator
Div. of Adolescent Medicine, University of Rochester
(585) 273-4616
or
Jonathan D. Klein, MD, MPHAssociate Professor of Pediatrics and of Community & Preventive Medicine University of Rochester
(585) 275-7760
Or visit...
The University of Rochester, Division of Adolescent Medicine, Leadership Education in Adolescent Health
website atwww.urmc.rochester.edu/gchas/div/adol/leah/resources.htm
or
The ACT for Youth, Center of Excellence website at www.actforyouth.net