Washington 21st Century Community Learning Centers Program ...
-
Upload
nguyendien -
Category
Documents
-
view
215 -
download
0
Transcript of Washington 21st Century Community Learning Centers Program ...
MONTH YEAR
Washington 21st Century
Community Learning Centers
Program Evaluation: Year 2
Neil Naftzger
Matthew Vinson
Feng Liu
Bo Zhu
Kimberly Foley
JANUARY 2014
Washington 21st Century
Community Learning Centers
Program Evaluation: Year 2
January 2014
Neil Naftzger
Matthew Vinson
Feng Liu
Bo Zhu
Kimberly Foley
20 North Wacker Drive, Suite 1231
Chicago, IL 60606-2901
312.288.7600 | Fax: 312.288.7601
www.air.org
Copyright © 2014 American Institutes for Research. All rights reserved. 14-1167_01/14
Contents Page
Executive Summary ......................................................................................................................... i
Program Quality ......................................................................................................................... i
Evaluation Questions and Methods ...........................................................................................v
Methods......................................................................................................................................v
Summary of Key Findings ....................................................................................................... vi
Chapter 1: Introduction ....................................................................................................................1
Evaluation Questions .................................................................................................................1
Reasoning for Chosen Evaluation Questions: Importance of Program Quality ........................1
Organization of Report ..............................................................................................................2
Chapter 2: Methods ..........................................................................................................................3
Data Sources and Analysis.........................................................................................................3
Chapter 3. Primary Characteristics of Washington 21st CCLC Programs and Participants ...........9
Grantee Characteristics ..............................................................................................................9
Center Characteristics ..............................................................................................................12
Summary of Grantee and Center Characteristics .....................................................................24
Chapter 4: Leading Indicators ........................................................................................................25
Overview of Leading Indicators ..............................................................................................25
Selected Leading Indicators .....................................................................................................26
Organization of Leading Indicators Chapter ............................................................................28
Organizational Context ............................................................................................................28
Summary of Organizational Context Findings and Recommendations ...................................36
Instructional Context ................................................................................................................36
Summary of Instructional Context Findings and Recommendations ......................................53
Mutually Reinforcing Context .................................................................................................54
Summary of Findings and Recommendations in Relation to the Community Context
Domain ...................................................................................................................................64
Youth Outcomes Leading Indicators .......................................................................................65
Determining Program Improvement Priorities From the Leading Indicator System .............68
Chapter 5: Assessing 21st CCLC Program Outcomes ...................................................................73
Within-Program and Impact Analyses .....................................................................................73
Within-Program Analyses ........................................................................................................73
Summary of Within-Program Analyses Findings ....................................................................87
Impact of 21st CCLC Participation on Student Achievement .................................................88
Summary of Impact Analyses Results .....................................................................................95
Conclusions ....................................................................................................................................97
References ....................................................................................................................................100
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—i
Executive Summary
Information summarized in this report is based on data collected and analyzed by American
Institutes for Research (AIR) and the David P. Weikart Center for Youth Program Quality
(Weikart Center) as part of a statewide evaluation of Washington 21st Century Community
Learning Centers (21st CCLC) programs. Results represent findings from Year 2 of a three-year
statewide evaluation. The purpose of this executive summary is to (1) set the context for the
evaluation design with regard to a primary focus on program quality, (2) outline the evaluation
questions and methods, and (3) summarize key findings within each of the identified evaluation
questions. To set the context for the evaluation design, a brief discussion on program quality,
AIR’s framework for understanding afterschool program quality, and the leading indicators of
afterschool program quality developed in collaboration with the Washington Office of
Superintendent of Public Instruction (OSPI) are provided. Following the discussion on program
quality, the evaluation questions and methods are outlined, and a summary of key findings within
each of the identified evaluation questions is presented.
Program Quality
Research on Program Quality
Program quality and the implementation of best practices supported by research are increasingly
recognized as pressing issues for the afterschool field (Granger, Durlak, Yohalem, & Reisner,
2007). Research on the impact of participation in afterschool programming on students’
academic and behavioral outcomes often produces mixed and inconclusive results (Granger,
2008). For example, three noteworthy meta-analyses of the impact of afterschool programming
found that a majority of the reviewed studies did not find better outcomes for afterschool
participants relative to nonparticipants (Durlak & Weissberg, 2007; Granger, 2008; Lauer et al.,
2006; Zief, Lauver, & Maynard, 2006). However, others found average positive effects of
afterschool program participation on students’ academic and behavioral outcomes, which they
attributed to subsets of higher quality programs driving the overall average positive student
outcomes (Durlak & Weissberg, 2007; Lauer et al., 2006). In short, average positive outcomes
across several afterschool programs were likely due to the effectiveness of a small number of
high-quality individual programs. These findings highlight a key relationship between the quality
of afterschool programming and the attainment of desired program outcomes.
Meaningful progress has been made relative to understanding how elements of program quality
support quality afterschool programs and the attainment of desired youth outcomes. For example,
a growing body of research suggests that program outcomes in the form of enhanced student
academic achievement outcomes are realized by delivering developmentally appropriate
programming that is grounded in core principles of youth development (Birmingham, Pechman,
Russell, & Mielke, 2005; Durlak & Weissberg, 2007). The delivery of developmentally
appropriate activities that align with principles of youth development varies as a function of staff
competencies, interpersonal skills, and knowledge (Vandell et al., 2005). Leading experts agree
that staffs’ ability to form meaningful personal staff-student relationships facilitates the delivery
of interactive and engaging program activities (Eccles & Gootman, 2002). Likewise, staff ability
to design and deliver developmentally appropriate and interactive program activities is likely to
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—ii
differ as a function of the level of support provided by the program overall. As noted by Smith
(2007), Glisson (2007), and Birmingham et al. (2005), a program climate that supports ongoing
staff reflection on and involvement in efforts to improve program quality is a key aspect of
effective youth-development programs. Programs characterized by a supportive and
collaborative climate encourage staff to engage in self-reflective practices that improve overall
program quality.
AIR Framework for Program Quality
The evaluation team at AIR has engaged in extensive work evaluating afterschool programs and
providing technical assistance to support high-quality programming. The framework outlined in
Figure I represents the most recent research related to the path to quality in afterschool programs
as well as the evaluation team’s collective expertise with afterschool programming. As shown in
Figure I, the achievement of desired youth outcomes is a function of complex interactions
between several program elements:
Youth Characteristics. These are the characteristics and contributions youth bring to the
afterschool setting that influence how they engage with and benefit from afterschool
programs.
Community Context. The resources and characteristics of the local and school
community context serve to support meaningful partnerships to develop program goals,
program design, and provide program guidance.
Program Participation. Youth are more likely to benefit from afterschool program
participation if they attend consistently, over a period of time, and participate in a variety
of activity types.
Program Quality. Program quality comprises a series of practices and approaches that
support the provision of developmentally appropriate, high-quality settings and activities
at the point of service. This includes practices and approaches adopted by (a) activity
leaders working directly with youth and (b) the organization as a whole, which provides
an infrastructure to support implementation of effective practice in the design, delivery,
and evaluation of afterschool programming.
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—iii
Figure I. AIR’s Quality Framework for Afterschool Programs
Leading Indicators of Afterschool Program Quality
To assess the extent to which centers implement research-supported best practices and
approaches, a set of newly defined leading indicators of afterschool program quality was
developed in collaboration with OSPI and the Weikart Center and discussed in the Year 1 report.
The leading indicators are meant to further complement programs’ participation in the Youth
Program Quality Improvement process, to provide additional information regarding how well
programs are progressing in implementing research-supported practices, and more importantly,
to identify areas in need of improvement.
A primary goal of the statewide evaluation was to provide 21st CCLC grantees with data to
inform program improvement efforts regarding their implementation of research-supported best
practices. AIR, the Weikart Center, and OSPI worked collaboratively to define a series of
leading indicators using data collected as part of the statewide evaluation. Specifically, the
leading indicator system was designed to do the following:
Summarize data collected as part of the statewide evaluation in terms of how well the
grantee and its respective centers are adopting research-supported best practices.
Allow grantees to compare their level of performance on leading indicators with similar
programs and statewide averages.
Facilitate internal discussions about areas of program design and delivery that may
warrant additional attention from a program improvement perspective.
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—iv
The leading indicator system is focused on quality program implementation as opposed to youth or
program outcomes. It is hypothesized and supported by research indicating that more consistent
implementation of research-supported best practices supports the attainment of desired youth
outcomes. During Year 2 of the evaluation, the leading indicators continued to be developed in order
to meet the goals in providing grantees with opportunities for program improvement.
The adopted leading indicators are organized into four overarching contexts: (1) Organizational
Context, which is focused on practices that occur among staff and management; (2) Instructional
Context, which is focused on practices that occur at the point of service, where staff and youth
directly interact; (3) Mutually Reinforcing Context, which is focused on practices related to
coordinating and aligning afterschool programming and activities with the regular school day,
family, and community contexts; and (4) Youth Outcomes Leading Indicators, which are focused
on the change in youths’ proficiency in reading/English language arts (ELA) and mathematics.
Organizational Context
Leading indicators within the Organizational Context examine both staff development and internal
communication and collaboration among program staff. Programs characterized by a supportive
and collaborative climate permit staff to engage in self-reflective practice to improve overall
program quality. Self-reflective practice is more likely to lead to high-quality program sessions that
provide youth with positive and meaningful experiences. Three leading indicators fall under the
Organizational Context: (1) Staff Capacity; (2) Continuous Improvement, which is assessed by
scales measuring program climate and internal communication and collaboration; and (3)
Leadership and Management.
Instructional Context
Leading indicators in the Instructional Context focus on the practices and approaches adopted by
frontline staff to design and deliver activity sessions that intentionally support youth skill
building and mastery that align with centers’ objectives and principles of youth development.
There are two leading indicators in the Instructional Context: (1) Quality of Instructional Content
and (2) Quality of Instructional Processes/Strategies.
Mutually Reinforcing Context
The Mutually Reinforcing Context focuses on relationships between the 21st CCLC program and
context external to the program that significantly impact the success of the program. Community
partners, families, and schools play an important role in the 21st CCLC programs by expanding
program activities, facilitating program sustainability, and providing important information about
student needs. Three leading indicators are associated with the Mutually Reinforcing Context:
(1) Family Engagement, (2) School Context, and (3) Community Context.
Youth Outcomes Leading Indicators
The Youth Outcomes Leading Indicators focus on whether students who regularly attend 21st
CCLC programming (defined as more than 30 days of attendance during the programming
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—v
period) shifted between state proficiency categories in reading/ELA and mathematics between
the 2010–11 and 2011–12 school years.
Evaluation Questions and Methods
Evaluation Questions
A key objective of the evaluation was to understand how well centers were implementing
research-supported best practices and approaches and to assess the impact of 21st CCLC
participation on students’ academic and behavioral outcomes. Specifically, the evaluation was
designed to answer the following evaluation questions:
1. What were the primary characteristics associated with the grants and centers funded by
21st CCLC and the student population served by the program?
2. To what extent is there evidence that centers funded by 21st CCLC implement research-
supported practices related to quality afterschool programming?
3. To what extent is there evidence of a relationship between center and student
characteristics and the likelihood that students demonstrated better performance on
program attendance and youth outcomes, with a particular emphasis on exploring the
relationship between leading indicator status and these outcomes?
4. To what extent is there evidence that students participating in services and activities
funded by 21st CCLC demonstrated better performance on youth outcomes as compared
with similar students not participating in the program?
Methods
Data Sources
To address the evaluation questions, the evaluation team collected data from the following
sources:
21st CCLC Profile and Performance Information Collection System (PPICS). Data
collected through the Annual Performance Report (APR) module of PPICS on grantee,
center, and student characteristics were extracted from PPICS.
Youth Outcome and Related Data From Comprehensive Education Data and
Research System (CEDARS). Academic and demographic information for 21st CCLC
participants and nonparticipants attending the same schools as 21st CCLC participants
were pulled from the CEDARS database.
Site Coordinator Survey. Site coordinators were surveyed about a variety of program
operations related to implementation of best practices in quality afterschool
programming.
Staff Survey. Program staff were surveyed about a variety of program operations related
to implementation of best practices in quality afterschool programming.
Youth Program Quality Assessment (YQPA) Data. Program staff completed surveys,
self-assessments, and observations as part of a quality assessment improvement program
to support grantees completing the Youth Program Quality Improvement (YPQI) process.
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—vi
Analysis
Descriptive analysis of PPICS data on grantee, center, and student characteristics along with cluster
analysis techniques were used to provide an overall description of Washington 21st CCLC
operating in the 2011–12 school year. Both descriptive analysis and Rasch analysis of site
coordinator and staff survey responses were used to assess the extent to which centers implement
research-supported best practices aligned with the previously described leading indicator system.
In order to group centers into clusters on the basis of their scores on leading indicator data,
hierarchical cluster analysis was used. Correlational multilevel modeling techniques were employed
to explore the relationship between student- and center-level characteristics associated with 21st
CCLC sites and youth outcomes. Finally, a propensity score matching approach was used to assess
the impact of 21st CCLC programming on youth outcomes by comparing participants with similar
nonparticipants from the same schools.
Summary of Key Findings
A summary of key findings within each of the identified evaluation questions is provided.
1. What were the primary characteristics associated with the grants and centers funded by
21st CCLC and the student population served by the program?
Grantee Characteristics
A total of 55 Washington 21st CCLC grantees were active during the 2011–12 school
year.
A majority of grantees (80 percent) were considered “mature” grants—not in the first or
last year of the five-year funding cycle.
Grantees were roughly split between the categories of school-based (53 percent) and non-
school-based (47 percent) grantee.
Center Characteristics
A total of 183 centers were in operation across the 55 active grantees for the 2011–12
school year.
A majority of centers (96 percent) were school based.
Centers mainly served children in elementary school (37 percent) and middle school (34
percent); 14 percent of centers served high school students.
Centers provided an average of 4.4 days of programming a week over eight months.
Roughly half of centers targeted students for enrollment due to students’ low
performance on local or state assessments.
A total of 3,029 staff members worked in centers for the 2011–12 school year.
Centers most commonly employed a mix of mostly school day teachers, other school
staff, and college students (42 percent), and mostly school day teachers (28 percent).
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—vii
A majority of centers offered mostly enrichment activities (45 percent) or a variety of
activities (27 percent).
A total of 24,379 students attended 21st CCLC programming for at least one day. Of the
total 21st CCLC participants, a majority (61 percent) were regular attendees (attended for
30 days or more).
On average, 21st CCLC regular participants attended 60 days of programming.
Overall, centers had approximately 82 regular attendees and 133 total attendees.
A majority of 21st CCLC participants were Hispanic (45 percent) or white (36 percent).
Most attendees (71 percent) qualified for free or reduced-price lunch, 19 percent were
classified as limited English proficient, and 11 percent were classified as special needs.
2. To what extent is there evidence that centers funded by 21st CCLC implement research-
supported practices related to quality afterschool programming?
As previously noted, leading indicators of afterschool program quality were developed to
examine how well centers implemented research-supported best practices. Findings related to
Evaluation Question 2 are summarized according to the overarching contexts for the leading
indicators and specific leading indicators within each context.
Organizational Context
Staff Capacity. Active participation in professional development and training is essential in
supporting staff capacity. Center staff were asked to report on the frequency and type of
professional development/training attended during the 2011–12 school year. A majority of staff
(63 percent) reported participating in some form of training. The most common topic of the
trainings attended by staff included strategies for delivering high-quality academic enrichment
activities and activities to support youth development.
Continuous Improvement. This leading indicator includes the following aspects of continuous
program improvement: (1) program climate, (2) internal communication as reported by site
coordinators, and (3) internal communication as reported by center staff. Key findings within
these aspects of continuous program improvement are summarized below.
Program Climate. The average scale score for program climate reported by center staff fell
within the agree response category (scale response categories included strongly disagree,
disagree, agree, and strongly agree), suggesting that most staff reported supportive,
collaborative program climates. A majority of centers (76 percent) fell in the agree response
category, and 22 percent fell in the disagree or strongly disagree response category. Center staff
were most likely to disagree with the following statements: (1) there is adequate time to plan
individual activity sessions and (2) staff participated fully in program decision making.
Internal Communication—Site Coordinator. For site coordinators, the average scale score for
internal communication fell within the a couple of times per year response category (response
categories included never, a couple of times per year, about once a month, and nearly every
week), suggesting that practices related to internal communication were implemented on an
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—viii
infrequent basis. A majority of site coordinator scale scores (65 percent) also fell in the a couple
of times per year response category.
Internal Communication—Center Staff. For center staff, average center scale scores fell
within the a couple of times per year response category. However, 27 percent of centers fell
within the a couple of times per year response category, and 57 percent of centers fell in the
about once a month response category. These results suggest that staff are slightly more likely to
engage in strategies for internal communication with other program staff as opposed to engaging
in internal communication strategies with their site coordinators. Center staff were least likely to
report using data to set program improvement goals with other staff, a shift from 2011–12 when
the least frequently implemented activity was observing other afterschool staff delivering
programming.
Leadership and Management. The average scale score reported by staff fell in the three
category (response categories included one, three, and five from the YPQA), suggesting that
most staff reported that the center’s leadership and management support youth-staff relationships
and a positive development focus, promote staff development, and are committed to ongoing
improvement. A majority of centers (77 percent) fell in the three category, and 10 percent fell in
the five response category.
Instructional Context
Quality of Instructional Content. Three separate scales were used to assess aspects of
programming related to the quality of instructional content: (1) alignment of program activities
with program objectives, (2) intentionality in program design as reported by site coordinators,
and (3) intentionality in program design as reported by center staff. Key findings within each of
these aspects of quality of instructional content are summarized below.
Alignment of Program Activities With Program Objectives. To assess the extent of alignment
between program objectives and program activities, site coordinators were asked to provide the
top three program objectives, and steps were taken to assess how frequently sites delivered
activities aligned with their top three program objectives. For example, it is expected that
programs identifying improving students’ grade-level proficiency as a top objective would spend
a significant amount of time on academic enrichment activities. From this analysis, results
indicated that a majority of centers (90 percent) delivered activities aligned with their identified
top three program objectives.
Intentionality in Program Design—Site Coordinator. Site coordinators were asked to report
how frequently staff leading program activities engaged in strategies reflective of intentional
program design. Average site coordinator scale scores fell in the frequently response category
(response categories included never, sometimes, frequently, and always), suggesting that site
coordinators felt practices related to intentional service delivery are commonly adopted by
activity leaders. Forty-five percent of site coordinator responses fell in the frequently category.
Intentionality in Program Design—Center Staff. Center staff members were asked to report
how frequently they engage in strategies indicative of intentional program design. The average
center scale score also fell in the frequently response category. A majority of centers (74 percent)
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—ix
fell in the frequently response category. This indicates that staff are slightly more likely to report
engaging in practices related to intentional program design relative to site coordinator reports of
how frequently staff engage in the same practices.
Quality of Instructional Processes/Strategies
Four separate scales were used to assess aspects of programming related to the quality of
instructional processes/strategies: (1) point of service quality, (2) youth-centered policies and
practices, (3) youth ownership according to the site coordinator survey, and (4) youth ownership
as reported on the staff survey. Key findings within each of these aspects of quality instructional
processes/strategies are summarized below.
Point of Service Quality. The average scale scores for the overall point of service quality fell
within the functioning near optimal category. Percentages of staff respondents stating that point
of service quality, safe environment, and supportive environment were functioning near optimal
were 81 percent, 100 percent, and 89 percent, respectively, although a majority of responses fell
in the still room for improvement category for interaction and engagement.
Youth-Centered Policies and Practices. The average scale scores for this leading indicator fell
within the three category (response options included one, three, and five), with a majority of
centers (76 percent) falling in this category and 10 percent falling in the one category. These
responses suggest that most staff report that programs tap youth interests, build youths’ skills,
and involve youth in the structure and policy of the program.
Youth Ownership—Site Coordinator. The average site coordinator scale score fell in the
disagree response category (response options were strongly disagree, disagree, agree, and
strongly agree). However, in looking at the distribution of site coordinator scale scores, a
majority (51 percent) of site coordinators fell in the agree category. There may be room for
growth in defining more organizational- or state-level strategies for cultivating youth ownership.
Opportunities for Youth Ownership—Center Staff. The average staff survey scale scores fell
in the agree response category. A majority of centers fell in the agree category (60 percent), and
37 percent fell within the disagree category.
Staff Capacity to Create Interactive and Engaging Settings. Staff were asked to rate the
collective capacity of frontline staff to create and provide interactive and engaging program
settings. Average staff scale scores fell in the agree category, suggesting that staff generally
agree that frontline staff adopt strategies likely to produce interactive and engaging settings.
With regard to the distribution of centers across response categories, a majority of centers (70
percent) fell in the agree response category, and 12 percent fell in the disagree category.
Service Delivery Practices. Questions on the service delivery practices scale asked staff to report
on practices they adopt in their own work with youth. The average staff scale score fell in the
available occasionally response category. Likewise, 56 percent of centers fell in the available
occasionally response category.
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—x
Mutually Reinforcing Context
The Mutually Reinforcing Context focuses on relationships between the 21st CCLC program and
context external to the program that significantly impacts the success of the program. Three
leading indicators are associated with the Mutually Reinforcing Context: (1) family engagement,
(2) school context, and (3) community context.
Family Engagement. Survey questions on the site coordinator survey assessed center
approaches to communicating with families. The average family communication scale score fell
within the sometimes response category (response options were never, sometimes, and
frequently), which is indicative of programs typically communicating with families once or twice
a semester. A majority (74 percent) of site coordinator responses fell in the sometimes response
category.
School Context. This leading indicator is meant to capture the degree to which 21st CCLC staff
members align the design and delivery of programming to the school day and individual student
needs. Survey questions related to linkages to the school day and data use were asked on the site
coordinator and staff surveys. The average site coordinator scale score fell within the minor
strategy response category for linkages to the school day, indicating that most sites employed
only a portion of strategies for establishing linkages with the school day. Fifty-three percent of
site coordinator respondents fell within the minor strategy response category, and 32 percent fell
within the major strategy category. For staff responses, the average scale score fell within the
disagree response category, suggesting that, on average, most staff have an incomplete sense of
both student academic needs and school day curriculum and/or instructions. Sixty-five percent of
centers fell in the agree response category, and 31 percent fell in the disagree category. The
average scale score for data use for both site coordinators and staff fell in the occasionally use
category, suggesting that the degree to which they use data is limited. Seventy-three percent of
site coordinators and 55 percent of staff responses fell into this category.
Community Context. The leading indicator for community context is meant to capture the
degree to which partners associated with the center are actively involved in planning, decision
making, evaluating, and supporting program operations, as well as the extent to which the
program adopts practices supportive of family and community engagement. The average site
coordinator scale score on the partner involvement scale fell within the do informally response
category (response options included did not do, do informally, and do formally). Generally,
although centers work with partners in many ways, they have a tendency to do so on an informal
basis as opposed to following formal policies and procedures. Of activities that site coordinators
engage in with partners, 19 percent do so formally, while 60 percent do so informally. It is also
important to note that 112 centers had partners that are actively involved in the provision of
programming directly to youth.
The average scale score for the family and community engagement scale, which is meant to
capture barriers to family and community involvement in the program, fell into the three
category (response options included one, three, and five). Seventy-seven percent of staff
respondents fell into this category, suggesting that a majority of staff reported policies at their
centers that promote family and community engagement in the program.
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—xi
Youth Outcomes
The leading indicator is meant to capture descriptively the extent to which students participating
in the program moved from one state proficiency category to another between the 2010–11 and
2011–12 school years. For example, 14 percent of regular attendees moved from the well below
standard category to the below standard category in the 2011–12 school year in mathematics.
3. To what extent is there evidence of a relationship between center and student
characteristics and the likelihood that students demonstrated better performance on
program attendance and youth outcomes, with a particular emphasis on exploring the
relationship between leading indicator status and these outcomes?
Correlational relationships between center- and student-level characteristics and youth outcomes
were explored in order to examine whether centers’ leading indicator status was related to
program attendance, academic performance, and unexcused absences. It was hypothesized that
there here would be a negative correlation between center membership in clusters where scores
on all leading indicators were below average and youth outcomes. Key findings include:
Membership in the Instructional Context Content Below Average cluster was negatively
associated with reading scores, as measured through students’ reading state assessments.
Membership in this cluster was also positively associated with unexcused absences. Each
of the findings was consistent with what was hypothesized.
Center membership in the Organizational Context Below Average cluster was negatively
associated with program attendance (this was consistent with what was hypothesized),
while displaying a positive relationship with credits earned and cumulative GPA (which
was not expected).
Membership in the Instructional Context Process Below Average cluster was found to be
significantly and positively related to program attendance. This is surprising but may be
related to the high number of elementary only programs in this cluster, as elementary
programs often have higher attendance than middle or high school programs.
The number of days of attendance in 21st CCLC programs was found to have a positive
relationship with academic performance-related outcomes.
4. To what extent is there evidence that students participating in services and activities
funded by 21st CCLC demonstrated better performance on youth outcomes as
compared with similar students not participating in the program?
Propensity score matching was employed to examine the impact of 21st CCLC programming on
participants as compared to nonparticipants with similar characteristics from the same schools.
Outcomes explored included academic performance and unexcused absences. Key findings are
summarized as follows:
Small but significant positive effects were found for reading and mathematics
achievement at both 30-day and 60-day participation levels when pooled across grades.
Students in the treatment group with 30-day participation achieved 0.027 standard
deviation units higher on reading and 0.044 standard deviation units higher on
mathematics than nonparticipants. For 60-day participation group, 21st CCLC
American Institutes for Research Washington 21st CCLC Program Year 2 Evaluation: Executive Summary—xii
participants scored 0.033 and 0.035 standard deviation units higher than nonparticipants
on reading and math, respectively.
There was a significant positive impact of the program on the cumulative GPA of
students with 60+ days of treatment; the cumulative GPA of this group was 0.195
standard deviation units higher than the comparison group. A significant positive effect
for this group was also seen for percentage of credits earned.
The participant group showed a statistically significant, negative impact of the 21st
CCLC programming on unexcused absences. Students in both the 30-day and 60-day
treatment groups had unexcused absences of 66 percent and 39 percent of the level in the
nonparticipant group, respectively.
Impacts varied across grade levels. For example, a significant negative impact on
cumulative GPA was found for students in the 60-day treatment group in Grade 9, and
there was a significant positive impact on students in Grades 10 and 11.
Recommendations
This report’s findings on leading indicators, correlational relationships, and impact analyses
provide guidance for grantees on areas for continued growth in the upcoming years, including (1)
using data to inform services for individual students, (2) allowing staff more time for planning and
preparation, and (3) identifying ways to incorporate more youth ownership into the program at
grade-appropriate levels. These results are very similar to those identified in the Year 1 report. In
addition, there appears to be some evidence that (a) there are opportunities for growth in terms of
how centers go about designing and delivering activities from a content perspective and (b) that
enhanced levels of practice in this area are related to better school-related outcomes. Although
OSPI has an infrastructure for supporting instructional quality from a process perspective, it may
want to give consideration to the types of supports it could provide to enhance the manner in which
21st CCLC programs support the cultivation of skills and knowledge from a content perspective,
particularly in relation to the needs of participating youth.
Although a variety of positive program effects were demonstrated in this year, OSPI is interested in
further exploring the types of impacts 21st CCLC is having on social-emotional learning, 21st
century skills and competencies, and noncognitive outcomes. Toward this end, in Year 3 of the
evaluation, AIR will be working to collect information from grantee project directors on what they
believe their programs are impacting in these areas and what their priorities should be in terms
testing measurement strategies to assess program impact on such outcomes. Steps will be taken to
select a sample of instruments designed to measure high-priority outcomes and pilot those in a
small number of centers during spring semester of the 2013–14 school year.
Finally, the leading indicators represent a substantial investment of time and effort to provide
Washington 21st CCLC grantees with actionable data to guide and support program improvement
efforts. A key goal of the Year 3 evaluation will be to better understand the efficacy of these tools
as a vehicle for supporting quality improvement efforts and to highlight portions of the system that
are proven to have especially high value to grantees and OSPI.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—1
Chapter 1: Introduction
Throughout the past nine years, 21st Century Community Learning Centers (21st CCLC) in the
state of Washington have provided afterschool and expanded learning programming to enhance
the academic well-being of students in high-poverty communities. This report highlights how
well afterschool programs (funded by 21st CCLC, subsequently referred to as centers)
throughout Washington have fared relative to meeting the goals and objectives for supporting
student growth and development as specified by the Washington Office of Superintendent of
Public Instruction (OSPI).
Information discussed in the following sections is based on data collected and analyzed by
American Institutes for Research (AIR) and the David P. Weikart Center for Youth Program
Quality (Weikart Center) as part of a statewide evaluation of Washington 21st CCLC programs.
The results represent findings from Year 2 of a three-year statewide evaluation, which will
conclude in January 2014.
Evaluation Questions
A key objective of the 2011–12 statewide evaluation of Washington 21st CCLC-funded
programming was to understand both how well centers were implementing programming in
terms of research-supported practices and approaches and what impact participation in 21st
CCLC-funded activities had on student academic outcomes. More specifically, the evaluation
was designed to answer the following set of evaluation questions:
1. What were the primary characteristics associated with both centers funded by 21st CCLC
and the student population served by the program?
2. To what extent was there evidence to suggest that centers funded by 21st CCLC had
adopted research-supported practices related to the provision of quality afterschool
programming?
3. To what extent is there evidence of a relationship between center and student
characteristics and the likelihood that students demonstrated better performance on
program attendance and youth outcomes, with a particular emphasis on exploring the
relationship between leading indicator status and these outcomes?
4. To what extent is there evidence that students participating in services and activities
funded by 21st CCLC demonstrated better performance on youth outcomes as compared
with similar students not participating in the program?
Reasoning for Chosen Evaluation Questions: Importance of Program Quality
Collectively, the domain of evaluation questions represents both the goals and objectives OSPI
has specified for the 21st CCLC program and emerging issues across the national landscape of
afterschool programming. For example, Granger (2008) notes that afterschool research often
demonstrates mixed and inconclusive results regarding the impact of participation in afterschool
programming on students’ academic and behavioral outcomes. Granger (2008) cites three
noteworthy meta-analyses of the impact of afterschool programming that found a majority of the
studies included in each meta-analysis did not find better outcomes for the afterschool participant
group relative to the comparison group (Durlak & Weissberg, 2007; Lauer et al., 2006; Zief,
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—2
Lauver, & Maynard, 2006). However, both Durlak & Weissberg (2007) and Lauer et al. (2006)
found average positive effects of participation in afterschool programming on students’ academic
and nonacademic outcomes, which they attributed to higher quality programs driving the overall
positive outcomes. In short, average positive outcomes across several afterschool programs were
likely due to the effectiveness of a small number of high-quality individual programs. These
findings highlight a key relationship between the quality of afterschool programming and the
attainment of desired program outcomes.
Although meaningful progress has been made in understanding elements of quality in afterschool
programming (e.g., Granger, Durlak, Yohalem, & Reisner, 2007; Little, 2007; Vandell et al.,
2005; Wilson-Ahlstrom & Yohalem, 2007; Yohalem, Wilson-Ahlstrom, Fischer, & Shinn,
2009), these understandings have largely been used to support the development of quality
assessment tools and improvement systems to help afterschool programs better understand (1)
criteria for afterschool program quality, (2) how well they measure up to identified criteria, and
(3) steps that can be taken to modify programming and enhance program quality. This reflects
the stance of leading researchers that the most pressing issue before the afterschool community is
developing effective quality-improvement systems (Granger et al., 2007).
OSPI, in collaboration with the Weikart Center, has taken steps to craft a quality assessment
improvement system and support grantees in completing the Youth Program Quality
Improvement (YPQI) process, which utilizes a youth development framework to combine self-
assessment, action planning, skill development, and targeted technical assistance to enhance
program quality. To address Evaluation Question 2 as noted previously, this year’s report
summarizes and expands the leading indicators introduced in the Year 1 report. The leading
indicators developed as part of the statewide evaluation are meant to further complement
programs’ participation in the YPQI process, to provide additional information regarding how
well they are progressing in implementing research-supported practices, and more importantly,
to identify areas of program operations in need of improvement. OSPI’s use of the YPQI process
and leading indicators provides 21st CCLC programs with an infrastructure to make data-driven
decisions about program improvement in a timely, meaningful, and systematic way. As
indicated, one of the goals of the statewide evaluation is to explore the relationship between
measures of program quality, as measured by the leading indicators, and student academic and
behavioral outcomes. Exploring this relationship is especially helpful in refining the leading
indicator system according to measures of program quality that relate to student outcomes.
Organization of Report
The following sections provide a summary of the methods, including data sources and analytic
techniques, to address the primary evaluation questions. Following an overview of the evaluation
methods, key grantee and center characteristics are summarized, with a particular emphasis on
characteristics that are considered key to improving student academic achievement and attaining
desired program outcomes. The leading indicator system is then summarized and explained with
regard to how information relates to future evaluation and technical assistance efforts. Finally,
analyses for assessing relationships between center- and student-level characteristics and student
outcomes, as well as for evaluating the impact of 21st CCLC participation on student-level
outcomes are summarized, along with conclusions and recommendations to guide future
evaluation and program improvement efforts.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—3
Chapter 2: Methods
Data Sources and Analysis
Data collected and analyzed in this report come from four primary sources, including
administrative data systems and surveys. Each data source and associated methods of data
analysis are described.
21st CCLC Profile and Performance Information Collection System (PPICS)
PPICS is a Web-based data collection system developed and maintained by AIR on behalf of the
U.S. Department of Education. Data on the full domain of 21st CCLC programs funded
nationally, including those in Washington, are collected through this system. Data collected
through the Annual Performance Report (APR) module of PPICS on center characteristics in
relation to the 2011–12 programming period were extracted from PPICS and utilized in several
analyses contained in this report, including information on program operations, staffing,
activities provision, and student attendance rates. A total of 184 programs associated with 55
active 21st CCLC grantees, during the 2011–12 programming period, were represented in the
data set extracted from PPICS. (Note: A single 21st CCLC grant typically has more than one
program associated with it.)
Youth Outcome and Related Data From CEDARS
AIR constructed a unique data collection module for Washington integrated within PPICS that
allowed for the collection of student-identifiable information that was extracted from the system
and provided to OSPI. OSPI used this information to perform a series of merges against state data
warehouses to obtain Measurements of Student Progress (MSP) reading and mathematics scores,
High School Proficiency Exam (HSPE) reading scores, cumulative GPA, credits earned, and the
number of unexcused absences, as well as additional demographic information about the students
in question from the Comprehensive Education Data and Research System (CEDARS), a
longitudinal data warehouse of educational data maintained by OSPI. OSPI also identified students
not participating in 21st CCLC programming who attended the same schools as 21st CCLC
participants and provided the same testing and related CEDARS information for these students.
These data were used both to conduct the impact analyses predicated on comparing 21st CCLC
participant with nonparticipant outcomes and to construct the set of models needed to explore the
relationship between center and student characteristics and student achievement and related
outcomes.
Site Coordinator Survey
An online survey of site coordinators working in 21st CCLC programs active during the 2011–12
school year was administered in spring 2012. The site coordinator was defined as the individual
at a given center who was responsible for the day-to-day operations of the program and was the
initial point of contact for parents and staff when questions or issues arose on-site. Generally, site
coordinators are seen as important middle managers in the delivery of 21st CCLC programming
at a given site.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—4
A total of 184 site coordinator surveys were administered. Completed surveys were received
from 173 site coordinators, for a response rate of 94 percent. The survey addressed the extent to
which centers engaged in practices that the research indicates are supportive of effective
afterschool programming. Sets of survey questions were organized to create scales measuring the
following dimensions of program operations:
Program objectives
Activity enrollment policies and recruitment approaches
Access to and use of student data
Linkages to the school day
Staffing approach and challenges
Other operational challenges
Intentionality in activity and session design
Creation of interactive and engaging settings for youth
Opportunities for youth ownership
Internal communication designed to support program development and improvement
Practices supportive of cultivating effective partnerships
Practices supportive of parent involvement and engagement
Professional development and training
Data obtained from the site coordinator surveys were used both to support the leading indicator
process and to construct variables included in analyses to assess the relationship between center
characteristics and 21st CCLC participant outcomes.
Staff Survey
The purpose of the online staff survey was to obtain information from frontline staff who worked
directly with youth during the 2011–12 school year. A particular focus of the survey was on
practices that support both positive academic outcomes and youth development outcomes. As
with the site coordinator survey, the staff survey included sets of questions associated with a
given scale, as well as open-ended questions to assess dimensions of program operations.
Dimensions of program operations assessed on the staff survey included the following:
Program objectives
Intentionality in activity and session design
Practices supportive of academic skill building, including linkages to the school day and
using data on student academic achievement to inform programming
Practices supportive of positive youth development
Opportunities for youth ownership
Internal communication designed to support program development and improvement
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—5
Program climate in terms of how staff view the organizational supports and structures as
supporting their work with youth
Training participation
Completed surveys were received from 1,090 center staff from 181 centers. The number of
completed staff surveys received per center ranged from one to 19, with an average of six
completed surveys per center. As with the site coordinator survey, data obtained from the staff
surveys were used to support the leading indicator process.
Youth Program Quality Assessment (YPQA) Data
As noted previously, OSPI, in collaboration with the Weikart Center, has taken steps to craft a
quality assessment improvement system and support grantees in completing the YPQI process.
As part of this process, observations were conducted by program staff as a self-assessment or by
trained external observers of activities provided by 21st CCLC grantees, and the YPQA Form A
was scored to provide an estimate of how safe, supportive, interactive, and engaging the
observed session was for participating youth. In addition, although the YPQA Form A is meant
to measure program quality at the point of service, the YPQA Form B is a rubric completed by
program staff on how well the program has adopted organizational processes that are likely to
engender and facilitate point of service quality. Both YPQA Form A and B data were uploaded
directly to the Weikart Center via the Center’s online score reporter.
It is important to note that participation in the YPQI process was voluntary for Washington 21st
CCLC grantees during the 2011–12 school year. As a result, PQA Form A data were available
for only 72 centers associated with 33 grantees. Form B was provided in relation to 62 centers
associated with 32 grantees.
Analytic Approach and Methods
It is important to note that the findings outlined in this report are primarily quantitative in nature.
This approach was largely driven by both the evaluation questions being answered and the
resources available to carry out the project during Year 2 of the project. Analyses highlighted in
this report fall within five general categories:
1. Descriptive Analyses. Information related to grantee, center, and student characteristics
obtained from PPICS, the surveys, and the PQA were analyzed descriptively to explore
the range of variation on a given characteristic. Some of the leading indicators also were
calculated employing descriptive analysis techniques.
2. Analyses to Create Scale Scores. Many questions appearing on the site coordinator and
staff surveys underpinning the leading indicators were part of a series of questions
designed to assess an underlying construct/concept, resulting in a single scale score
summarizing performance on a given area of practice or facet of 21st CCLC afterschool
implementation (e.g., practices that support linkages to the school day). An example is
shown Figure 1, which outlines the questions making up the Intentionality Program
Design scale that appeared on the site coordinator survey.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—6
Figure 1. An Example of a Survey Scale Calibrated Using Rasch Techniques
For scales such as this, Rasch scale scores were created using staff and site coordinator
responses to a series of questions to create one overall score. These scale scores ranged
from 0 to 100, where higher scores were indicative of a higher level or more frequent
adoption of a specific quality practice or set of practices.
Scale scores resulting from the application of Rasch approaches can also be used to
classify what portion of the rating scale the average scale score fell within. For example,
the statewide mean value for the Intentionality in Program Design scale highlighted in
Figure 1 was 59.82, which put the statewide average in the frequently range of the scale
indicating the typical staff member responding to the survey reported engaging in these
practices on a frequent basis. This approach also allowed the evaluation team to explore
the distribution of centers in light of what response option their average scale score put
them in.
The primary benefit of this approach is the capacity to distill responses from several
questions down into one overall score for the center, simplifying the process of
interpreting how a center did on a given element of quality, particularly in relation to
other programs in the state.
3. Hierarchical Cluster Analysis. Hierarchical cluster analysis was employed to combine
centers into groups based on how well they scored on the leading indicator data collected
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—7
during the 2011–12 school year. Cluster analysis is typically employed to combine cases
(or, in this case, centers) into groups using a series of variables as criteria to determine
the degree of similarity between individual cases and is particularly well suited when
there is a desire to classify a large number of cases into a smaller domain of discrete
groupings. Employing this approach allowed the evaluation team to synthesize the full
domain of leading indicator data into a series of more discrete and meaningful quality
profile types, making it easier to describe how centers active during the 2011–12 school
year performed relative to the indicators overall and to create variables that could more
easily be added to the multilevel models described below.
4. Correlational Multilevel Modeling Techniques. Several multilevel models were run to
explore the relationship between center-level and student-level characteristics associated
with sites funded by 21st CCLC and student-level outcomes, including attendance in 21st
CCLC programs and performance on state assessments in reading and mathematics and
other school-related outcomes. Although these analyses afford the capacity to say if a
significant relationship existed between a center- or student-level characteristic and a
given outcome such as mathematics achievement, these approaches cannot indicate that a
given characteristic caused a given outcome. In this sense, these analyses are
correlational, but not causal, in nature.
5. Propensity Score Matching. In contrast to the multilevel modeling techniques,
propensity score matching approaches were employed to estimate the causal impact of
21st CCLC participation on student performance in reading and mathematics using MSP
and HSPE scores obtained from OSPI, as well as a series of other school-related
outcomes. Given that 21st CCLC program participants were not randomly assigned to
participate in the program, the problem of selection bias was an issue that needed to be
addressed before program impact could be explored from a causal perspective. It is likely
that students who participated in 21st CCLC programming were different from those
students attending the same schools who do not enroll in 21st CCLC. These differences
can bias estimates of program effectiveness because they make it difficult to disentangle
preexisting differences between participants and nonparticipants from program impact.
Propensity score matching was used to mitigate that existing selection bias in program
effect.
Table 1 provides a summary of the methods that were employed to answer each evaluation
question.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—8
Table 1. Summary of Methods by Evaluation Question
Evaluation Question Descriptive
Analysis
Rasch
Analysis
Hierarchical
Cluster
Analysis
Correlational
Multilevel
Modeling
Propensity
Score
Matching
What were the primary characteristics associated with
the grants and centers funded by 21st CCLC and the
student population served by the program?
To what extent was there evidence that centers funded
by 21st CCLC implement research-supported
practices related to quality afterschool programming?
To what extent is there evidence of a relationship
between center and student characteristics and the
likelihood that students demonstrated better
performance on program attendance and youth
outcomes, with a particular emphasis on exploring the
relationship between leading indicator status and these
outcomes?
To what extent is there evidence that students
participating in services and activities funded by 21st
CCLC demonstrated better performance on youth
outcomes as compared with similar students not
participating in the program?
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—9
Chapter 3. Primary Characteristics of Washington 21st
CCLC Programs and Participants
One of the hallmarks of the 21st CCLC program is the wide diversity (1) of organizations
involved in the provision of 21st CCLC programming, (2) of approaches to the manner in
which services and activities are delivered, and (3) in the nature of the student population
served. In this chapter, the primary characteristics associated with both grantees and centers
funded by 21st CCLC and the student population served by the program are outlined in relation
to the 2011–12 programming period.
Grantee Characteristics
OSPI is responsible for distributing 21st CCLC funds it receives from the U.S. Department of
Education via a competitive bidding process that results in the awarding of new grants to
entities that propose to operate centers in high-poverty communities. Grants active during the
2011–12 programming period were initially awarded in 2007 (n = 11), 2008 (n = 12), 2009
(n = 21), and 2010 (n =11). (No grants were reported with an award date in 2011.) The term
grantee in this report refers to an entity that applied for and received a 21st CCLC grant from
OSPI, serving as the fiscal agent for the grant in question. This section considers elements that
can be considered only at the grant level, notably grant maturity, grant organization type, and
first-year award amounts. Where feasible, an effort has been made to compare Washington
grantees with all grantees nationwide active during the 2011–12 reporting period tracked in
PPICS.
Grantee Maturity
Grantee maturity was examined as part of the evaluation because of the following hypothesis:
Based on their experience, more mature centers have found ways to provide higher quality
services, adapt more readily to budget reductions, and have planned to sustain the programs after
the grant funding ends. To facilitate comparisons with national data housed in PPICS,
Washington grantees were classified into three possible maturity categories:
New: grantees in their first year of 21st CCLC funding
Mature: grantees not in their first year, but also not in their last year of funding
Sustaining: grantees in their last year of 21st CCLC funding
As shown in Table 2, among Washington grantees active during the 2011–12 programming
period, the vast majority were found to fall in the mature category (80 percent), and the
remaining grants were in their last year of operation (20 percent). Grants were given for a five-
year period.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—10
Table 2. Grants by Maturity
WA Grants
All Grants Nationwide*
Grant Maturity
N Grants
% Grants
N Grants**
% Grants
New 0 0.0% 421 10.6%
Mature 44 80.0% 2,072 52.1%
Sustaining 11 20.0% 1,974 37.3%
Total Grantees 55 100.0% 3,974 100.0%
*Note. As stated in Chapter 2, the national numbers were not finalized at the time of compiling this report;
four states were still incomplete. These numbers therefore reflect the vast majority of grantees nationwide,
but not all.
**Organization maturity could not be determined for 142 grantees at the national level.
Grantee Organization Type
As established in the authorizing legislation for 21st CCLC, programs may be administered by
several types of grantee agencies. The most relevant distinction is whether or not the grantee
organization is a school-based entity. School-based organizations (SBO) include school districts,
charter schools, and private schools. Non-school-based organizations (NSBO) include, among
other entities, community-based organizations, faith-based organizations, health-based
organizations, and park districts.
Of the 21st CCLC grantees funded by Washington, school-based and non-school-based
organizations have been represented roughly equally since the state-administered program began.
During the course of the 2011–12 programming period, for example, school districts were the
fiscal agents on 29 of the 55 active grants (53 percent of all 21st CCLC grants). Figure 2 shows
the comparison across six APR years.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—11
Figure 2. School-Based Versus Non-School-Based Grantees
Of the non-school-based grantees, Regional/Intermediate Educational Agencies are the largest
group, making up more than 18 percent of all grantees in 2012, substantially higher than what is
the case nationwide. The next highest non-school-based grantee type was community-based
organizations, making up approximately 16 percent of all fiscal agents, which is comparable with
national norms.
Grant Amounts
Washington’s first-year grant award amounts and the duration of the grants were assessed
alongside national averages, as shown in Table 3. No major differences in terms of the average
length of a grant were noted between the two groups, although the average first-year award for
Washington grantees was somewhat lower than the national average. The median first-year
award amounts for Washington and the nation (Washington inclusive) were, respectively,
$220,000 and $200,000, indicating a smaller number of very large grants is driving the national
average to a higher amount.
13 13 17
20 21 26 26
16 16 14
16
25
29 29
0
10
20
30
40
50
60
2006 2007 2008 2009 2010 2011 2012
School Based
Non-School Based
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—12
Table 3. Grants by First-Year Award Amount*
WA Grants
All Grants Nationwide
Award Amount and Duration
Mean
Mean
Year 1 award amount $271,259 $329,451
Award length 5 4.5
Total grantees
Mean number of centers per grant
55
3.3
4,116
2.5**
*Of grantees reporting data for APR 2012.
**Exclusive Washington Grants.
Center Characteristics
One of the primary goals of this report is to examine the relationship between key center
characteristics and the likelihood that centers will have a positive impact on student achievement
and behavioral outcomes. It is important to note that in this report, the term center is used to refer
to the physical location where 21st CCLC-funded services and activities take place. Centers are
characterized by defined hours of operation, have dedicated staffs, and usually have positions
akin to site coordinators. Each 21st CCLC grantee in Washington has at least one center; many
grantees have more than one center. During the course of the 2011–12 reporting period, there
were a total of 183 centers providing 21st CCLC-funded activities and services.
In addition, center characteristics can be termed either to be indicative of research-supported best
practices or simply as innate attributes of the center in question without a strong connection to
the afterschool quality practice literature. Center characteristics indicative of the latter might
include the grade level served, program maturity, and organizational type. For example,
identifying a program as one that serves only elementary students says nothing about the quality
of that program. Although these types of variables are included in models oriented toward
assessing the impact of the program on desired student outcomes, this report does not focus on
them in depth.
Other characteristics, such as the activity (e.g., mostly tutoring, mostly academic enrichment)
and staffing model employed, at a site are still somewhat ambiguous when viewed from a quality
practice standpoint, with the literature less clear on the superiority of certain activities or staffing
approaches. Some preliminary results derived from the PPICS data set seem to show certain
advantages in these areas (i.e., mostly tutoring programs and program staffed by school day
teachers), but the manner in which these data are collected and processed do not lend themselves
to robust casual inferences about the viability of one approach instead of another. Similar
analyses conducted as part of other statewide evaluations have produced more ambiguous results
in terms of how these characteristics may be related to student outcomes. The analyses contained
in this report are intended to build an understanding of whether certain activity or staffing
models seem to be more often correlated with positive youth outcomes and thereby warrant
consideration as a quality practice worthy of emulation and replication. As with the
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—13
characteristics detailed earlier, however, this report does not spend a great deal of time exploring
them from a purely characteristic standpoint.
Finally, the domain of characteristics assessed through the site coordinator and staff surveys are
meant to clearly reflect the best-practices literature. Particular attention will be dedicated in this
report to explaining how staff responded to site coordinator and staff survey questions and what
this response may mean in terms of how programs design and deliver activities in ways that are
consistent with best practices. These results are highlighted in particular in the section dedicated
to explaining the newly adopted leading indicator system.
Center Organization Type
Just as with grants, centers can be classified as either school-based or non-school-based. During
the 2011–12 programming period, approximately 95.6 percent of Washington’s centers were
located in schools. This percentage is a little more than the national average of 86.7 percent.
Figure 3. School-Based Versus Non-School-Based Centers
School-Year and Summer Operations
In terms of periods of operation, Washington centers tended most often to offer programming
after the school day (as opposed to before the school day, during the school day, or on
weekends), offering on average 10.3 hours of programming after school each week. On average,
Washington offered slightly less programming during the school year than did centers across the
nation, with roughly 12.1 hours of programming per week compared with a national average of
13.4 hours per week. Washington centers offered programming an average of 4.4 days per week
over 32 weeks, which is similar to the national averages for the 2011–12 programming period.
152 145 142
155 165
181 175
6 4 2 5 7 4 8
0
20
40
60
80
100
120
140
160
180
200
2006 2007 8 2009 2010 2011 2012
School Based
Non-School Based
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—14
In terms of summer operations, a total of 121 of Washington’s centers (66.1 percent) offered
summer programming. This was an increase from previous years: The percentage of centers with
summer programs was 55.1 percent in 2006, 62.4 percent in 2007, 45.1 percent in 2008, 48.8
percent in 2009, 34.3 percent in 2010, and 59.5 percent in 2011. In this regard, in 2012
Washington centers were slightly more likely than other centers nationwide to offer summer
programming (with a national average of 53.9 percent). Otherwise, Washington centers tended to
be very similar to other centers nationwide in terms of summer operation averages. Washington
centers with summer programs had, on average, 5.0 weeks of programming (compared with 5.3
weeks nationally) and approximately 19 hours of programming per week (compared with 25
hours of programming per week nationally). Overall, Washington centers are fairly typical for
the nation in terms of program operation.
Center Staffing
The quality of center staffing is crucial to the success of afterschool programming (Vandell et al.,
2005), and many of the program improvement approaches being used in the field emphasize the
importance of staff for creating positive developmental settings for youth. The success of
afterschool programs is critically dependent on students forming personal connections with the
staff—especially for programs serving older students, where a much wider spectrum of activities
and options is available to youth (Eccles & Gootman, 2002).
Similar to their counterparts nationally, Washington 21st CCLC programs employ a variety of
staff, including academic teachers, nonacademic teachers, college and high school students,
counselors, paraprofessionals from the school day, youth development workers, and other
program staff with a wide spectrum of backgrounds and training. A total of 3,029 staff members
were reported for 2011–12 school year operations (33.6 percent volunteer) and 1,081 for the
summer of 2011 (31.2 percent volunteer). Of the school year staff, 24.9 percent were paid school
day teachers. Another 13.0 percent were paid staff with a college degree. Volunteer high school
students were the largest volunteer group, accounting for 9.9 percent of school year staff.
Summer staffing was very similar to school year staffing in terms of relative configuration, with
28.7 percent of summer staff being paid school day teachers, and 11.0 percent being other paid
staff with a college degree. Volunteer community members accounted for 6.8 percent of all
summer staff.
To summarize the different staffing models used by programs active during the 2011–12
programming period, centers were classified into groups or clusters based on the extent to which
they relied on different types of staff to deliver activities, using cluster analysis techniques.1 Data
used to construct these clusters were obtained from PPICS. Figure 4 presents the five primary
staffing models that were identified in the programs.
Based on this analysis, Washington has a relatively high percentage of centers classified as
(a) Mostly School Day Teachers and Other School Staff as well as (b) Mostly School Day
Teachers, the two most common staffing clusters at the national level.
1 Cluster analysis is typically employed to combine cases into groups using a series of variables as criteria to
determine the degree of similarity between individual cases. Cluster analysis is particularly well suited to a study in
which there is a desire to classify a large number of cases into a smaller domain of discrete groupings.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—15
Figure 4. Staffing Clusters, Washington and the Nation (2012 APR)
Note. Based on 183 centers in Washington and 9,560 centers nationally with complete staffing information.
Center Activities
Both the staff working at a given 21st CCLC and the activities offered to students attending the
program in question are critical elements in how youth experience and potentially benefit from
their participation in 21st CCLC. Nationally, the goal of the 21st CCLC program is to provide
academic and nonacademic enrichment programs that reinforce and complement the regular
academic program of participating students. This overarching charge is broad and encompasses a
host of different types of activities, including the following types that are tracked in PPICS:
Academic enrichment learning program
Recreational activity
Homework help
Supplemental Education Services tutoring
Activity to promote youth leadership
Expanded library service hours
Drug/violence prevention, counseling, or character education
Career/job training
Promotion of family literacy
Mentoring
Community service/service learning
17.3%
38.2%
2.6%
30.7%
11.1%
16.7%
27.8%
2.2%
42.2%
11.1%
0%
10%
20%
30%
40%
50%
60%
70%
YD, Oth No Coll,
SD Teach
SD Teach Oth, SD Teach SD Teach, Oth
School Staff
College Stu, SD
Teach
All States
Washington
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—16
Promotion of parent involvement
Other (e.g., activities involving computers and technology, life skills, nutrition, etc.)
In order to further classify centers into categories that meaningfully represent the relative
emphasis given to providing different types of activities (academic enrichment, tutoring,
homework help, recreation, etc.), K-Means cluster analysis also was employed using center-level
percentages for each category of activity. When compared with the nation, centers in
Washington were more likely to fall into the Enrichment cluster (45 percent of all centers
compared with 23 percent of centers nationally) or the Variety cluster (with 27 percent of all
centers in Washington, compared with 34 percent nationally). See Figure 5.
Figure 5. Activity Clusters, Washington and the Nation (2011–12 Programming Period)
Note. States have the option to require their centers to submit activities data in the APR in one of two
different ways: as aggregated hours or as individual activity records. Because only individual activity
records are used to carry out the cluster analysis in question, the numbers presented under “Activity
Cluster” represent centers in states that opted to employ the individual activity record option. For all
states, there were 4,541 centers with individual activity cluster designations (Washington inclusive); for
Washington, there were 155 centers with individual activity cluster designations.
Grade Levels Served
A topic garnering increasing attention at the national level relates to the role that grade level
plays, both in terms of how 21st CCLC programs should structure their operations and program
activities and the outcomes for which they should be accountable through performance indicator
systems. Using student-level data about the grade level of students attending a program, 21st
CCLC programs were classified as follows:
Elementary Only: Centers serving students up to Grade 6
23.9%
9.4%
33.8%
20.7%
12.2%
7.7% 10.3%
26.5%
45.2%
10.3%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
Recreation Tutoring Variety Enrichment Homework Help
All States
Washington
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—17
Elementary/Middle School: Centers serving students up to Grade 8
Middle School Only: Centers serving students in Grades 5–8
High School Only: Centers serving students in Grades 9–12
Other: Centers that did not fit one of the other five categories
The High School Only category is especially important to examine because afterschool programs
for older children often look considerably different from elementary or middle school programs
(Naftzger et al., 2007). High school students’ needs are different from younger students, and they
often have other afternoon obligations such as jobs or extracurricular activities. In terms of grade
levels served, centers in Washington most commonly served elementary school students
exclusively, with 37 percent of all centers being classified as Elementary Only during the 2011–
12 programming period. However, as Figure 6 shows, starting during the 2008–09 programming
period, centers serving middle school age youth became increasingly common, representative of
an OPSI-initiated policy shift to fund more programs serving middle and high school age youth.
Figure 6. Percentage of Centers per Grade-Level Cluster, per Year
Note. Reflective of 183 centers with grade-levels-served status available.
Center Attendance
Attendance is an intermediate outcome indicator that reflects the potential breadth and depth of
exposure to afterschool programming. In this regard, attendance can be considered in terms of
(1) the total number of students who participated in the center’s programming throughout the
course of the year and (2) the frequency and intensity with which students attended programming
when it was offered. The former number can be utilized as a measure of the breadth of a center’s
reach, and the latter can be construed as a measure of how successful the center was in retaining
students in center-provided services and activities.
54
%
9%
14
%
4%
4%
14
%
57
%
11
%
15
%
4%
4%
9%
47
%
11
%
26
%
6%
6%
5%
38
%
7%
36
%
4%
10
%
5%
38
%
4%
35
%
6%
15
%
2%
38
%
4%
35
%
6%
15
%
2%
37
%
6%
34
%
8%
14
%
2%
0%
10%
20%
30%
40%
50%
60%
70%
Elem Elem-Mid Mid Mid-High High Other
2006 2007 2008 2009 2010 2011 2012
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—18
As part of the APR data-collection process in PPICS, information was collected on the total
number of students that a given center served during the programming period; how many of
those students met the definition of regular attendee by participating in 30 or more days of
programming; and demographic information about the student population in question, including
grade level and ethnicity.
In Washington, a total of 24,379 students were reported as attending 21st CCLCs for at least one
day during the 2011–12 programming period. Of these, 14,966 were regular attendees (students
who attended a total of 30 days or more during the reporting period), or 61.4 percent (compared
with 49.9 percent nationally). Attendance levels year-over-year are presented in Figure 7. The
decline in attendance levels between 2009 and 2010 is representative of an OSPI-adopted policy
change that increased the number of days a student would need to attend in order to be counted
as a participant.
Figure 7. Attendees and Regular Attendees in Washington State, by APR Year
Nearly half of students who met the definition of regular attendee participated in 21st CCLC-
funded activities for 30 to 39 days, with a steady decline in the number of students attending with
each increasing 10-day attendance band. See Figure 8.
9,708 9,167 9,426 12,304
13,877 14,951 14,966
13,502 13,276 10,022
15,627
6,608
8,829 9,413
0
5,000
10,000
15,000
20,000
25,000
30,000
2006 2007 2008 2009 2010 2011 2012
Non-Regular Attendees (Students)
Regular Attendees (Students)
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—19
Figure 8. Number of Students by Number of Days Attended
Overall, the mean school year attendance rate for regular attendees was 60 days, with a median
of 53. For summer, the mean attendance rate for regular attendees was 15 days, with a median of
14 days. On average, each center in Washington had approximately 133 total students and 82
regular attendees. This was about the same as total attendance in APR 2011. Median values show
a similar trend. See Figure 9 for year-over-year trends.
Figure 9. Average Attendance Rate per Center by APR Year,
Total and Regular Attendees (Washington Only)
In terms of ethnicity, Washington centers mostly served Hispanic and white students, with 45
percent of all regular attendees being Hispanic and 36 percent of regular attendees identified as
4,213
2,290
1,876 1,586
1,203 978
847 650
486 307
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
30-39 40-49 50-59 60-69 70-79 80-89 90-99 100-109 110-119 >= 120
To
tal
Nu
mb
er o
f R
egu
lar
Att
end
ees
Total Days Attended (Summer 2011 and School Year 2012)
146.9 150.6
135.1
174.6
119.1 128.5 133.2
61.4 61.5 65.5 76.9 80.7 80.8 81.8
0
20
40
60
80
100
120
140
160
180
200
2006 2007 2008 2009 2010 2011 2012
Total Students (Avg)
Regular Attendees (Avg)
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—20
being white. See Figure 10 for more detail on the number of students served in Washington by
ethnic group.
Figure 10. Number of Total Students and Regular Attendees, by Ethnicity
The 21st CCLC facilities have been specifically designed to provide afterschool activities and
services to students living in high-poverty communities. Typically, student eligibility for free and
reduced-price lunch is the metric relied upon to assess how well states and grantees are reaching
this target population. As shown in Figure 11, roughly 71 percent of all attendees and 75 percent
of regular attendees were eligible for free and reduced-price lunch (FRPL) during the 2011–12
programming period.
Figure 11. Number of Total and Regular Attendees, by FRPL Status
Note. The number of students whose FRPL status was unknown is not shown.
10,745
8,894
1,468 1,318 1,409
6,616
5,293
997 827 917
0
2,000
4,000
6,000
8,000
10,000
12,000
Hispanic White Black Asian Native American
Total Attendees
Regular Attendees
23,210 22,443
19,448
27,931
20,485
23,780 24,379
9,708 9,167 9,426
12,304 13,877
14,951 14,966
12,917 15,016 13,233 18,490 13,801 16,133 17,301 5,269 6,615 7,034 9,107 10,081 10,675 11,196 0
5,000
10,000
15,000
20,000
25,000
30,000
2006 2007 2008 2009 2010 2011 2012 2006 2007 2008 2009 2010 2011 2012
Total Attendees Regular Attendees
Total Students
FRPL Unknown
FRPL
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—21
In addition to FRPL eligibility, additional information about the student population served by
21st CCLC recorded in PPICS includes students designated as being limited English proficient
(LEP) and as having special needs. In 2011–12, 19 percent of all students and 21 percent of
regular attendees were LEP students, and 11 percent of all attendees and 11 percent of regular
attendees were classified as having a special need of some sort. Additional information about
each of these subgroups is outlined in Figures 12 and 13.
Figure 12. Number of Total and Regular Attendees, by Limited-English-Proficiency Status
Note. The number of students whose LEP status was unknown is not shown.
Figure 13. Number of Total and Regular Attendees, by Special-Needs Status
Note. The number of students whose special-needs status was unknown is not shown.
23,210 22,443
19,448
27,931
20,485
23,780 24,379
9,708 9,167 9,426
12,304 13,877
14,951 14,966
3,809 4,636 4,150 4,896 3,474 3,824 4,555 1,670 2,194 2,221 2,817 2,746 2,853 3,143 0
5,000
10,000
15,000
20,000
25,000
30,000
2006 2007 2008 2009 2010 2011 2012 2006 2007 2008 2009 2010 2011 2012
Total Attendees Regular Attendees
Total Students
LEP Unknown
LEP
23,210 22,443
19,448
27,931
20,485
23,780 24,379
9,708 9,167 9,426
12,304 13,877
14,951 14,966
1,373 1,936 1,774 2,669 2,117 2,457 2,605 645 813 955 1,295 1,437 1,683 1,690 0
5,000
10,000
15,000
20,000
25,000
30,000
2006 2007 2008 2009 2010 2011 2012 2006 2007 2008 2009 2010 2011 2012
Total Attendees Regular Attendees
Total Students
Spec Needs Unknown
Spec Needs
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—22
Enrollment Policies and Recruitment Approaches
Enrollment policies and recruitment practices may have a substantial bearing on program design
and delivery. For example, a program that targets a relatively small number of students with high
academic needs and proposes to provide them with intensive support in one-on-one and small-
group settings will have different strategies for recruitment and enrollment from a program that
aims to serve as many students as possible and provide those students with a rich array of
academic and nonacademic enrichment activities. Questions related to each of these areas were
asked on the site coordinator survey administered in the spring of 2012.
In terms of enrollment policies, site coordinators were asked to indicate the degree to which
activities provided at their site were:
Open to all students who want to participate
Only able to support limited enrollment and therefore filled on a first-come, first-served
basis
Based on giving enrollment priority to certain groups of students
Restricted in that only certain groups of students are eligible to participate
As Figure 14 shows, 51 percent of responding site coordinators indicated that all of the activities
provided at their site were open to all students who wanted to participate, and another 20 percent
indicated that most of their activities were open to all students. Clearly, the vast majority of the
centers active during the 2011–12 programming period provided activities that were largely open
to all students who wanted to participate. In contrast, only 10 percent of centers indicated that all
of the activities provided at their site were restricted in that only certain groups of students were
eligible to participate, and another 9 percent indicated that most of the activities they provided
were restricted.
Figure 14. Summary of Enrollment Policies Reported by Site Coordinators
51%
13% 16% 10%
20%
8% 16%
9% 15%
36%
23% 22% 15%
44% 45%
59%
0%
20%
40%
60%
80%
100%
Activities are open to allstudents that want to
participate
Activities are only able tosupport limited
enrollment and aretherefore filled on a firstcome, first served basis
Activities are based ongiving enrollment priority
to certain groups ofstudents
Activities are restricted inthat only certain groups
of students are eligible toparticipate
Re
spo
nd
ing
Site
Co
ord
inat
ors
All of the activities at this site
Most of the activities at this site
Some of the activities at this site
None of the activities at this site
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—23
In terms of recruitment approaches, site coordinators were asked a series of questions regarding
the extent to which students served at their site were recruited for enrollment in the program
based on the following:
The fact that the student scored “below proficient” on local or state assessments
The fact that the student failed to receive a passing grade during a preceding grading
period
A referral from school day staff because the student needed additional assistance in
reading or mathematics
The student’s status as an English language learner (ELL)
As Figure 15 displays, 43 percent of responding site coordinators indicated that most of the
student were enrolled in the program given that they had scored “below proficient” on local or
state assessments, and a majority of site coordinators indicated that some of the students had
been directed to the program because they failed to receive a passing grade during the preceding
grading period, were referred directly by school day staff, or were classified as an ELL student.
Figure 15. Summary of Enrollment Policies Reported by Site Coordinators
11%
43% 39%
7% 4%
25%
57%
14%
7%
32%
57%
5% 2%
11%
53%
34%
0%
20%
40%
60%
80%
100%
All of the students enrolledat this site
Most of the studentsenrolled at this site
Some of the studentsenrolled at this site
None of the studentsenrolled at this site
Re
spo
nd
ing
Site
Co
ord
inat
ors
The fact that the student scored “below proficient” on local or state assessments
The fact that the student failed to receive a passing grade during a preceding grading period
A referral from school day staff because the student needed additional assistance in reading ormathematics
The student’s status as an English Language Learner (ELL)
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—24
Summary of Grantee and Center Characteristics
Generally, the domain of Washington 21st CCLC grantees and centers operating during the
2011–12 reporting period were largely similar to grantees and centers nationwide in terms of
organizational and operational characteristics, although some differences were noted, as follows:
Washington grants were less likely to be held by school districts and more likely to
be held by regional/intermediate educational agencies (called educational service
districts in Washington) than in the nation as a whole.
Washington centers were less likely be staffed mostly by school day teachers.
Washington centers were more likely to adopt a mostly academic enrichment
program model when delivering activities.
It is not immediately clear if any significance should be attached to these differences between
Washington 21st CCLC grantees and the nation as a whole. Hypothetically, non-school-based
entities and programs less reliant on school day teachers may experience some additional
challenges in connecting activities to the school day or, at least at a minimum, may need to take
additional steps to ensure the necessary mechanisms to support communication and collaboration
are put in place. This theme will be explored more thoroughly in the leading indicator chapter
that follows as some of the leading indicators adopted for the 2011–12 programming period
pertain to the issue of linking 21st CCLC programming to the school day.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—25
Chapter 4: Leading Indicators
Overview of Leading Indicators
A primary goal of the statewide evaluation was to provide 21st CCLC grantees with data to
inform program improvement efforts regarding their implementation of research-supported best
practices. AIR, the Weikart Center, and OSPI worked collaboratively to define a series of leading
indicators predicated on data collected as part of the statewide evaluation. The leading indicators
were meant to enhance existing information/data available to 21st CCLC grantees regarding how
they fare in the adoption of program strategies and approaches associated with high-quality
afterschool programming. Specifically, the leading indicator system was designed to do the
following:
Summarize data collected as part of the statewide evaluation in terms of how well the
grantee and its respective centers are adopting research-supported best practices.
Allow grantees to compare their level of performance on leading indicators with similar
programs and statewide averages.
Facilitate internal discussions about areas of program design and delivery that may
warrant additional attention from a program improvement perspective.
The leading indicator system is focused on quality program implementation as opposed to youth or
program outcomes. It is designed to feed existing data (from PPICS) and program evaluation data
back to programs regarding the adoption of research-supported practices, so programs can identify
strengths and weaknesses and reflect on areas of program design and delivery in need of further
growth and development. Figure 16 provides an overall depiction of the intention, purpose, and
process of the leading indicator system. More consistent implementation of research-supported
best practices will theoretically support the attainment of desired youth outcomes.
It is important to note that the indicators presented in this report are based on an initial attempt to
develop a leading indicator system for Washington 21st CCLC grantees, and it is anticipated that
the system will be refined and developed in future years. Although these measures are drawn
from the research literature, the evidence base linking performance on these particular measures
with the achievement of desired student outcomes is limited. In addition, many of the measures
are based on self-reported data and perceptions of program implementation provided by 21st
CCLC staff. As such, results should be treated with caution and not utilized to draw definitive
conclusions about the quality, approaches, and practices adopted by centers during 2011–12
operating period.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—26
Figure 16. The Leading Indicator Process
Selected Leading Indicators
The nine adopted leading indicators are organized into four overarching contexts: (1) Organiza-
tional Context, focused on practices that occur among staff and management; (2) Instructional
Context, focused on practices that occur at the point of service, where staff and youth directly
interact; (3) Mutually Reinforcing Context, focused on practices related to coordinating and
aligning afterschool programming and activities with the regular school day, family, and
community contexts; and (4) Youth Outcome Leading Indicators, focused on the change in
students’ proficiency in reading/English language arts (ELA) and mathematics. Leading
indicators within each of these contexts are listed in Table 4.
Table 4. Leading Indicators by Context
1. Organizational Context
Leading Indicator 1.1 Staff Capacity
Leading Indicator 1.2 Continuous Improvement
Leading Indicator 1.3 Leadership and Management
2. Instructional Context
Leading Indicator 2.1 Quality of Instructional Content
Leading Indicator 2.2 Quality of Instructional Processes/Strategies
3. Mutually Reinforcing Context
Leading Indicator 3.1 Family Engagement
Leading Indicator 3.2 School Context
Leading Indicator 3.3 Community Context
4. Youth Outcome Leading Indicators
Leading Indicator 4.1 Reading/ELA and Mathematics Performance
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—27
Each of the adopted contexts and indicators are representative of AIR’s larger framework for
understanding the path to quality in afterschool programs. The achievement of desired youth
outcomes is a function of a complex set of interactions between several program elements:
Youth Characteristics. The characteristics and contributions youth bring to the afterschool
setting influence how they engage with and benefit from afterschool programs.
Community Context. The resources and characteristics of the local and school
community context serve to support meaningful partnerships to develop program goals,
program design, and provide program guidance.
Program Participation. Youth are more likely to benefit from afterschool program
participation if they attend consistently, over a period of time, and participate in a variety
of activity types.
Program Quality. Program quality is a series of practices and approaches that support
the provision of developmentally appropriate, high-quality settings and activities at the
point of service. This includes practices and approaches adopted by (a) activity leaders
working directly with youth (such practices are represented in the Instructional Context
domain in the leading indicator system) and (b) the organization as a whole, which
provides an infrastructure to support implementation of effective practice in the design,
delivery, and evaluation of afterschool programming (represented in the Organizational
Context and Mutually Reinforcing Context domains in the leading indicator system).
The current iteration of the leading indicator system addresses only a portion of the quality
framework depicted in Figure 17; there are a number of opportunities to expand the leading
indicator system to more fully represent additional, important components of afterschool
program quality.
Figure 17. AIR’s Quality Framework for Afterschool Programs
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—28
Organization of Leading Indicators Chapter
This chapter is organized, first and foremost, by the four broad contexts included in the leading
indicators. Within each context, data associated with a leading indicator in a given context are
summarized (for Washington centers overall). Two primary approaches to summarizing state-
level leading indicator data were used, as follows:
Scaled Items. Many questions on the site coordinator and staff surveys are part of a
series of questions designed to assess an underlying construct/concept and result in a
single scale score summarizing performance on a given aspect of a leading indicator (e.g.,
practices that support linkages to the school day). For these scale scores, Rasch scale
scores were created using staff and site coordinator responses to a series of survey
questions to create one overall score. Indicators analyzed using Rasch scales include a
scale score ranging from 0 to 100, where higher scores are indicative of a higher level or
more frequent adoption of a leading indicator. Average scale scores and the distribution
of scale scores across the response categories for a given scale are provided. For example,
a mean value of 56.57 may put the statewide average for a given indicator in the agree
range of the scale with response options for strongly disagree, disagree, agree, and
strongly agree). Site coordinator scale scores represent responses from one site
coordinator, and center scale scores represent the average of scale scores for all staff
respondents associated with a given center.
Descriptive Items. Other leading indicators are based on data that are not appropriate for
the type of scale construction just described. For example, program objectives are stand-
alone items that do not necessarily contribute to an underlying construct or concept. Items
of this type are summarized descriptively.
Organizational Context
Leading indicators within the Organizational Context examine both staff development and
internal communication and collaboration among program staff. As noted by Smith (2007),
Glisson (2007), and Birmingham, Pechman, Russell, and Mielke (2005), an organizational
climate that supports staff in reflecting on and continually improving program quality is a key
aspect of effective youth development programs. Programs characterized by a supportive and
collaborative climate permit staff to engage in self-reflective practice to improve overall program
quality. Self-reflective practice is more likely to lead to high-quality program sessions that
provide youth with positive and meaningful experiences. Three leading indicators fall under the
Organizational Context: (1) Staff Capacity; (2) Continuous Improvement, which is assessed by
scales measuring program climate and internal communication and collaboration; and (3)
Leadership and Management.
Leading Indicator 1.1: Staff Capacity
This leading indicator is meant to capture the degree to which staff receive training on delivering
high-quality instruction and are supported by middle and upper management in their efforts to
participate in professional development and training. Staff were asked a series of survey
questions related to the types of training they participated in during the 2011–12 school year.
As shown in Table 5, 63 percent of respondents reported participating in some type of training
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—29
related to their role in the afterschool program. It should be noted that there was a difference in
terms of the most frequent types of training reported by staff between the 2010–11 and 2011–12
school years. As shown in Table 6, while the training for providing effective academic
enrichment activities remained as one of the most frequent types, the other one shifted from
providing activities to support youth development to maintaining healthy and safe environment.
Although data were not available to explore if participation in the YPQI process influenced these
results, it is assumed that participation in the YPQI initiative had some level of impact on staff
responses.
Table 5. Staff Responses to Questions about Training Participation
PROMPT: Which of the following types of training were required and/or offered to you
during the present school year, and which did you attend or do you plan to attend in the
future? Please check all that apply.
Statewide
(N = 989)
% of Responders Reporting Participation in 21st CCLC Training 63%
Delivering effective enrichment activities. 61%
Providing activities to support youth development. 58%
Maintaining healthy and safe environments. 61%
Providing academic content in an afterschool setting. 46%
Learning how to apply principles related to child and adolescent growth and
development to activity design and delivery. 41%
Conflict resolution and behavior management. 50%
Working with a diverse student population. 49%
Parent and family engagement. 42%
Using data on student needs to inform programming. 34%
Providing activities that support college and workforce readiness. 18%
Other 5%
Data Source: Staff Survey
Table 6. The Most Frequent Types of Training Reported by Staff
Type of Training 2010-2011 School Year 2011-2012 School Year
Providing effective academic
enrichment activities √ √
Providing activities to support youth
development √
Maintaining healthy and safe
environment √
Data Source: Staff Survey
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—30
Leading Indicator 1.2: Continuous Improvement
Data for this leading indicator are summarized with Rasch scale scores ranging from 0 to 100,
where higher scores are indicative of higher levels of performance or endorsement of a given
scale. Three Rasch scale scores were calculated for this indicator to summarize the following
aspects of continuous improvement:
Program Climate: The extent to which program staff report that a supportive and
collaborative climate exists within the program (from the staff survey)
Internal Communication—Site Coordinator: How frequently site coordinators engage
in practices that support internal staff communication and collaboration (from the site
coordinator survey)
Internal Communication—Staff: How frequently staff engage in internal
communication and collaboration (from the staff survey)
Program Climate
Scale scores for program climate are based on the following question from the staff survey:
PROMPT: Please rate the extent to which you agree or disagree with the following with respect
to climate in your program:
There is adequate time to focus on individual student needs within the program time
frame.
The program staff has shared control over the content.
The staff is encouraged to try new and innovative approaches.
Instructional collaboration among program staff is encouraged and supported.
Staff is provided with training in current research on best practices in afterschool
programs.
Staff participate fully in program decision making.
There is adequate time to plan individual activity sessions.
As shown in Table 7, the statewide average scale score on the program climate fell within the
agree range of the scale (scale response options included strongly disagree, disagree, agree, and
strongly agree), suggesting that most staff reported supportive, collaborative program climates.
This conclusion is echoed in Figure 18, summarizing the distribution of centers in the four
response categories. A majority of centers (75.56 percent) fell in the agree response category,
and 21.67 percent fell in the disagree or strongly disagree response category. It should be noted
that there was a difference regarding the two statements staff were most likely to disagree with
between the 2010–11 and 2011–12 school years. As shown in Table 8, although There is
adequate time to plan individual activity sessions remained as one of the two statements for both
programming periods, the other one shifted from Staff is provided with training in current
research on best practices in afterschool programs to Staff participate fully in program decision
making. In these instances, there are ways OSPI can better support afterschool staff. For
example, future requests for proposal (RFPs) can be modified to require that programs build in
time for session planning or offer and support staff participation in program decision making.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—31
Table 7. Statewide Performance on the Program Climate Scale
Statewide Mean
Response Category
for Statewide Mean
Program Climate The extent to which program staff report that a supportive
and collaborative climate exists within the program.
59.71
(N = 1053)
Agree
(N = 1053)
Data Source: Staff Survey
Table 8. Statements That Staff Are Most Likely To Disagree With
Question 2010–11 School Year 2011–12 School Year
There is adequate time to plan
individual activity sessions √ √
Staff is provided with training in
current research on best practices in
afterschool programs
√
Staff participated fully in program
decision-making √
Data Source: Staff Survey
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—32
Figure 18. Distribution of Centers in Response Categories for Survey Questions About
Program Climate
Source: Staff Survey (1,053 responses from 180 centers)
Internal Communication
Scale scores of internal communication included staff and site coordinator responses to the
following survey question:
PROMPT: How often do you engage in the following tasks with other staff working in the program?
Conduct program planning based on a review of program data with other staff.
Use data to set program improvement goals with other staff.
Discuss progress on meeting program improvement goals with other staff.
Observe other afterschool staff delivering programming in order to provide feedback on
their practice.
Conduct program planning with other staff in order to meet specific learning goals in
coordinated ways across multiple activities.
Share ideas with other staff on how to make programming more engaging for
participating students.
Share experiences and follow up about individual youth and other staff.
Engage in discussions with other staff and school day teachers and/or administrators on
how the program could better support student learning needs.
1.67%
20.00%
75.56%
2.78%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Strongly Disagree Disagree Agree Strongly Agree
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—33
Participate in training and professional development with other staff on how to better
serve youth.
Discuss current research-based instructional practices with other staff.
The average statewide scale score for internal communication fell within the a couple of times
per year response category for site coordinators (scale response options included never, a couple
of times per year, about once a month, and nearly every week), suggesting the assessed
collaborative efforts were relatively infrequently implemented during the 2011–12 programming
period. For staff, the statewide average scale score also fell within the a couple of times per year
response category (see Table 9).
As shown in Figures 19 and 20, 64.71 percent of site coordinators fell in the a couple of times
per year response category, and 26.52 percent of centers fell within the same response category.
Staff survey respondents fell in the about once a month response category, with 59.67 percent of
centers falling in this response category (see Figure 20). These results may suggest that staff
members are slightly more likely to engage with one another in types of internal communication
assessed by the scale as opposed to engaging in internal collaboration with their site
coordinators. For staff, the least frequently implemented internal communication activity was to
Use data to set program improvement goals with other staff, although it was Observe other
afterschool staff delivering programming in order to provide feedback on their practice in 2010–
11 programming period (see Table 10). Yet, we anticipate these results may vary across sites
enrolled in the YPQI initiative versus those that were not.
Table 9. Statewide Performance on the Internal Communication Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Internal Communication—Site Coordinator Survey The frequency with which the site coordinator engages in
practices with program staff that support internal
communication and collaboration.
59.26
(N = 185)
A couple of times
per year
(N = 185)
Internal Communication—Staff Survey The frequency with which the staff engages in practices
with other program staff that support internal
communication and collaboration.
59.75
(N = 1,026)
A couple of times
per year
(N = 1,026)
Data Source: Site Coordinator and Staff Surveys
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—34
Figure 19. Distribution of Site Coordinators in Response Categories for Survey Questions
About Internal Communication
Source: Site Coordinator Survey (185 responses)
Figure 20. Distribution of Centers in Response Categories for Survey Questions About
Internal Communication Based on Staff Survey Responses
Source: Staff Survey (1,026 responses from 180 centers)
2.35%
64.71%
29.41%
3.53%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Never A Couple of Times
Per Year
About Once a Month Nearly Every Week
Sit
e C
oo
rdin
ato
rs
3.31%
26.52%
59.67%
10.50%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Never A Couple of Times
Per Year
About Once a Month Nearly Every Week
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—35
Table 10. The Least Frequently Implemented Internal Communication Activity
Internal Communication Activity 2010–11 School Year 2011–12 School Year
Use data to set program improvement
goals with other staff. √
Observe other afterschool staff
delivering programming in order to
provide feedback on their practice.
√
Data Source: Staff Survey
Leading Indicator 1.3: Leadership and Management
This leading indicator is meant to capture the degree to which the program has taken steps to hire
qualified staff, promote staff development, support program improvement, and solicit feedback.
Some of these areas overlap with previously identified indicators in the Organizational Context
domain, but the data presented in relation to this indicator directly represent how the program
believes it is doing in carrying out leadership and management tasks. This indicator is based on
data obtained from the Form B of Youth Program Quality Assessment (YPQA), a validated
instrument designed to evaluate the quality of youth programs and identify staff training needs.
The YPQA Form B focuses on program quality at the organizational level and assesses the
quality of organizational supports for the youth program offering assessed in Form A.
Staff were asked a series of questions regarding staff availability and longevity with the center,
qualifications, staff development, and ongoing program improvement. As shown in Table 11, the
statewide average scale score for leadership and management fell within the three range of the
scale (scale response options included one, three, and five), suggesting that most staff reported
the leadership and management in the center support youth-staff relationships and a positive
development focus, promote staff development, and are committed to ongoing program
improvement. This conclusion is echoed in Figure 21, indicating a majority of centers (77.42
percent) fell in the three response category, and 9.68 percent fell in the five response category.
Table 11. Statewide Performance on the Leadership and Management Scale
Statewide Mean
Response Category
for Statewide Mean
Leadership and Management – YPQA Form B
The extent to which the program is engaging in
practices that ensure staff are well positioned to
create developmentally appropriate settings for youth
and that processes are in place to support program
improvement efforts
52.43
(N = 62)
Three
(N = 62)
Data Source: YPQA Form B
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—36
Figure 21. Distribution of Centers in Response Categories for Survey Questions About
Leadership and Management
Source: YPQA Form B
Summary of Organizational Context Findings and Recommendations
As previously noted, the leading indicator system is part of a larger infrastructure constructed by
OSPI to support 21st CCLC-funded program improvement. This larger infrastructure includes the
YPQI quality improvement process. During the course of the 2011–12 programming period,
roughly half of active centers participated in the YPQI initiative on a voluntary basis. Although not
formally examined, it is hypothesized that YPQI sites were more apt to report engaging in the
types of practices and approaches described in the Organizational Context leading indicators.
In light of this, it is recommended that OSPI consider mandating participation in a YPQI-like
process for 21st CCLC grantees during some point of their five-year grant period. From a policy
perspective, OSPI also may want to consider making modifications to future 21st CCLC RFPs to
articulate this requirement and include standard budget line items where sites can identify the
resources they will dedicate to staff participation in YPQI-like processes. This will help ensure
that the value and importance of quality monitoring activities are relayed to programs and that
programs secure the necessary resources to effectively participate in such efforts.
Instructional Context
Leading indicators in the Instructional Context focus on the practices and approaches adopted by
frontline staff to design and deliver activity sessions that intentionally support youth skill
building and mastery that align with the center’s objectives and principals of youth development.
There is a strong connection between the leading indicators in the Instructional Context and
components of the YPQI program improvement process. For example, the YPQI process
12.90%
77.42%
9.68%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
One Three Five
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—37
assesses and supports staff practices at the point of service related to creating safe, supportive,
interactive, and engaging environments. The benefits of intentional design are also reflected in
the work of Durlak and Weissberg (2007), who found that effective afterschool programs
commonly provided activities that were sequenced, involved active forms of learning, and
focused on cultivating particular skills. There are two leading indicators in the Instructional
Context: (1) Quality of Instructional Content and (2) Quality of Instructional Processes/
Strategies.
Leading Indicator 2.1: Quality of Instructional Content
This leading indicator is meant to capture the degree to which the time spent on activities
corresponds to program objectives as identified by site coordinators and how intentionally
activities are designed and delivered. Both descriptive and Rasch scaling approaches were used in
relation to these data. Three separate metrics were calculated to describe aspects of this indicator:
Program Objectives: The degree to which site coordinators’ top three objectives align
with the proportion of time spent on corresponding activities
Intentionality in Program Design—Site Coordinator Survey: The frequency with which
staff engages in practices that indicate intentionality in activity and session design for the
delivery of activities meant to support student growth and development in reading and
mathematics
Intentionality in Program Design—Staff Survey: The frequency with which staff
engages in practices that indicate intentionality in activity and session design for the
delivery of activities meant to support student growth and development
Program Objectives and Alignment With Time Dedicated to Corresponding Activities
In order to assess alignment between activity provision and the program objectives identified by
the center in question, site coordinators were asked to rank their top three program objectives. In
order to assess alignment, steps were then taken to define each objective in regard to the proportion
of total activity time that could be minimally dedicated to particular activities to meet the identified
program objective. For example, if a site coordinator indicated that a primary program objective
was to enable low-performing students to achieve grade-level proficiency, then it was expected that
a certain number of hours would be dedicated to providing activities designed to support skill
building in core academic areas. As shown in Table 12, on average, 90 percent of center objectives
aligned with the actual frequency of activities provided during the programming period.
As shown in Table 12, the most common top three program objectives included (1) raise the
academic performance levels of any students who have an interest in participating (79 percent
endorsing), (2) provide students with access to academic enrichment opportunities (64 percent
endorsing), and (3) enable low-performing students to achieve grade-level proficiency (43 percent
endorsing). Based on these results, it is clear that most 21st CCLC site coordinators understand the
importance of prioritizing student academic growth and development and providing activities that
align with this priority.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—38
Table 12. Statewide Performance on the Program Objectives Aligned to Activity Provision
PROMPT: Please indicate which of these program objectives
constitute the top three priorities for your program. Statewide
(N = 188)
Program Objectives—the degree to which site coordinators’ top
three objectives align with proportion of time spent on
corresponding activities.
90%
Percentage of Site Coordinators Indicating Objective Was
Among Their Top Three Priorities
Enable low-performing students to achieve grade-level
proficiency. 43%
Provide students with access to academic enrichment
opportunities. 64%
Raise the academic performance levels of any students who
have an interest in participating. 79%
Provide supervised space for students to complete homework. 21%
Enhance the social or civic development of students. 29%
Prepare students for college and work. 29%
Provide students with the opportunity to participate in sports and
recreation activities. 21%
Enhance the artistic development of students (e.g., visual and
performing arts, etc.). 7%
Other 7%
Data Source: Site Coordinator Survey and PPICS
Intentionality in Program Design
As previously noted, a growing body of research suggests that program outcomes in the form of
enhanced student academic achievement outcomes are realized by simply paying attention to
how programming is delivered—specifically, whether or not programming is delivered in
developmentally appropriate settings grounded in core principles of youth development
(Birmingham et al., 2005; Durlak & Weissberg, 2007). In addition to youth development
principles, afterschool programs are more likely to attain desired student academic outcomes if
staff members responsible for planning the content of sessions incorporate certain practices and
strategies into their planning efforts.
On both the site coordinator and staff surveys, a series of questions was asked about intentional
program design.
Scale scores for intentionality in program design included staff and site coordinator responses to
the following survey questions:
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—39
PROMPT: How often do staff lead activities that are especially meant to support student growth
and development in reading and/or mathematics and provide program activities that are ...
Based on written plans for the session, assignments, and projects?
Well planned in advance?
Tied to specific learning goals?
Meant to build upon skills cultivated in a prior activity or session?
Explicitly meant to promote skill building and mastery in relation to one or more state
standards?
Explicitly meant to address a specific developmental domain (e.g., cognitive, social,
emotional, civic, physical, etc.)?
Informed by the express interests, preferences, and/or satisfaction of participating youth?
Although the items appearing on each survey were the same, site coordinators were asked to
indicate how frequently staff leading activities to support skill building in reading and/or
mathematics engaged in the practices listed above, and staff were asked how frequently they
engaged in these practices. It should be noted that some differences between site coordinator and
staff responses to the above survey questions may be associated with the fact that staff who are
not responsible for leading activities that support skill building and mastery in reading and
mathematics also completed surveys and were included in the analysis.
As shown in Table 13, the average site coordinator scale scores fell within the frequently
response category (response options were rarely, sometimes, frequently, and always), suggesting
that site coordinators felt practices related to intentional service delivery are commonly adopted.
Average staff scale scores also fell in the frequently response category. As Figures 22 and 23
show, 44.71 percent of site coordinator scale scores fell in the frequently response category, and
73.53 percent of centers fell in the frequently response category. This indicates that staff is more
likely to report engaging in practices related to intentional program design relative to site
coordinator responses of how frequently staff engage in practices related to intentional program
design.
It is possible that differences between site coordinator and staff responses suggest that some staff
are acting in a more autonomous fashion when planning activities, operating outside of any
organizational structures or criteria for planning activity sessions. Generally, this is an area that
warrants additional attention by OSPI, particularly in reference to the previously discussed
program climate findings that a substantial proportion of frontline staff struggle to find adequate
time to plan activity sessions and offerings.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—40
Table 13. Statewide Performance on the Intentionality in Program Design Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Intentionality in Program Design—Site Coordinator
Survey The frequency with which staff engage in practices that
indicate intentionality in activity and session design among
staff responsible for the delivery of activities meant to
support student growth and development.
59.82
(N = 185)
Frequently
(N = 185)
Intentionality in Program Design—Staff Survey The frequency with which staff engage in practices that
indicate intentionality in activity and session design among
staff responsible for the delivery of activities meant to
support student growth and development.
59.64
(N = 1,051)
Frequently
(N = 1,051)
Data Source: Site Coordinator Survey
Figure 22. Distribution of Site Coordinator Scale Scores From Survey Questions About
Intentional Program Design
Source: Site Coordinator Survey (185 responses)
8.24%
36.47%
44.71%
10.59%
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
Rarely Sometimes Frequently Always
Sit
e C
oo
rdin
ato
rs
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—41
Figure 23. Distribution of Center Scale Scores From Survey Questions About Intentional
Program Design Based on Staff Survey Responses
Source: Staff Survey (1,051 responses from 180 centers)
Leading Indicator 2.2: Quality of Instructional Processes/Strategies
This leading indicator is meant to capture the processes and practices in which staff members
engage that are consistent with high-quality instruction and core youth development principles,
with particular emphasis on providing developmentally appropriate activities at the point of
service. Conceptually, many of the practices associated with this indicator are related to the
concepts embedded in YPQA. All of the data reported in relation to this indicator were scored
using Rasch scale scores ranging from 0 to 100, where higher scores are indicative of higher
performance/higher frequency on the assessed aspects of leading indicator 2.2. Six separate scale
scores were calculated to assess aspects of this leading indicator:
Point of Service Quality—YPQA Form A: The extent to which program staff provide
supports and opportunities to create safe, supportive, interactive, and engaging settings
for participating youth
Youth-Centered Policies and Practices—YPQA Form B: The extent to which the
program adopts youth-centered policies and practices conducive to a supportive learning
environment
Youth Ownership—Site Coordinator Survey: The extent to which the site coordinators
perceive that program staff provide opportunities to develop youth ownership in the
program
Youth Ownership—Staff Survey: The extent to which program staff perceive the
presence of opportunities to develop youth ownership
2.35%
22.94%
73.53%
7.06%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Rarely Sometimes Frequently Always
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—42
Capacity to Create Interactive and Engaging Settings: The extent to which staff
members perceive that programming provides interactive and engaging settings for
participating youth
Service Delivery Practices: The frequency with which staff adopt specific practices that
support youth development
Point of Service Quality
This leading indicator is assessed by scales measuring safety, supportive environment,
interaction, and engagement. Table 14 displays both self-assessment and/or external assessment
data obtained by scoring the YPQA Form A observational tool. Scores were placed on a 0 to 100
scale and were adjusted to account for the bias introduced by the type of assessor (i.e., external
or self-assessment) and the type of activity observed (i.e., enrichment, tutoring/homework help;
or recreation).
As shown in Table 14, the average scale scores for the safe environment and the supportive
environment fell within the functioning near optimal category, and those for the interaction and
engagement fell within the still room for improvement category. But the average statewide scale
score for overall point of service quality fell within the functioning near optimal category.
This conclusion is echoed in Figures 24 through 28. For point of service quality, safe
environment and supportive environment, the percentages of staff respondents that fell within the
functioning near optimal category are 80.56 percent, 100 percent, and 88.89 percent,
respectively. In contrast, the percentages of staff respondents that fell within the still room for
improvement category for interaction and engagement are 83.33 percent and 90.28 percent.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—43
Table 14. Statewide Performance on the Point of Service Quality Scale
Statewide Mean
Response Category
for Statewide Mean
Point of Service Quality – YPQA Form A
The extent to which program staff provide supports and
opportunities to create safe, supportive, interactive, and
engaging settings for participating youth (total YPQA
score).
53.55
(N = 219)
Functioning Near
Optimal
(N = 219)
Safe Environment 68.48
(N = 219)
Functioning Near
Optimal
(N = 219)
Supportive Environment 57.35
(N = 219)
Functioning Near
Optimal
(N = 219)
Interaction 39.25
(N = 219)
Still Room for
Improvement
(N = 219)
Engagement 33.06
(N = 219)
Still Room for
Improvement
(N = 219)
Data Source: Form B PQA
Figure 24. Distribution of Centers in Response Categories for Survey Questions About
Point of Service Quality
Source: YPQA Form B
80.56%
19.44%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
Functioning Near Optimal Still Room for Improvement
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—44
Figure 25. Distribution of Centers in Response Categories for Survey Questions About
Safe Environment
Source: YPQA Form B
Figure 26. Distribution of Centers in Response Categories for Survey Questions About
Supportive Environment
Source: YPQA Form B
100.00%
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
Functioning Near Optimal
Cen
ters
88.89%
11.11%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Functioning Near Optimal Still Room for Improvement
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—45
Figure 27. Distribution of Centers in Response Categories for Survey Questions About
Interaction
Source: YPQA Form B
Figure 28. Distribution of Centers in Response Categories for Survey Questions About
Engagement
Source: YPQA Form B
16.67%
83.33%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
Functioning Near Optimal Still Room for Improvement
Cen
ters
9.72%
90.28%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Functioning Near Optimal Still Room for
Improvement
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—46
Youth-Centered Policies and Practices
This leading indicator is meant to capture the degree to which the program adopts youth-centered
policies and practices conducive to a supportive learning environment. The data presented in
relation to this indicator are based on data obtained from the YPQA Form B. Staff were asked a
series of questions about the program’s relevance to youth interests and skills, as well as youths’
influence on the setting, activities, structure, and policy of the center. As shown in Table 15, the
statewide average scale score for youth-centered policies and practices fell within the three
range of the scale (scale response options included one, three, and five). Figure 29 shows that a
majority of centers (75.81 percent) fell within the three category, while 9.68 percent fell in one
category. This indicates that most staff reported programs tap youth interests, build multiple
skills, and involve youth in the settings, activities, structure and policy of the program.
Table 15. Statewide Performance on the Youth-Centered Policies and Practices Scale
Statewide Mean
Response Category
for Statewide Mean
Youth-Centered Policies and Practices – YPQA Form B
The extent to which the program adopts youth-centered
policies and practices and practices conducive to a
supportive learning environment.
52.42
(N = 62)
Three
(N = 62)
Data Source: YPQA Form B
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—47
Figure 29. Distribution of Centers in Response Categories for Survey Questions About
Youth-Centered Policies and Practices
Source: YPQA Form B
Youth Ownership
Youth ownership refers to allowing youth to shape the afterschool program by setting goals for
what they want to accomplish in the program, making choices about both the content and process
of delivering offerings, planning activities, and having a role in governing the program.
Scale scores for youth ownership include staff and site coordinator responses to the following
survey question:
PROMPT: Please indicate your level of agreement with the following statements about how your
students build ownership of the program:
Youth are afforded opportunities to take responsibility for their own program.
Youth have the opportunity to set goals for what they want to accomplish in the program.
Youth help make plans for what activities are offered at the program.
Youth make choices about what content is covered in program offerings.
Youth make choices about how content is covered in program offerings.
Youth help create rules and guidelines for the program.
9.68%
75.81%
14.52%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
One Three Five
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—48
As shown in Table 16, the average site coordinator scale scores fell in the disagree category, and
staff survey scale scores fell in the agree response category (response options were strongly
disagree, disagree, agree, and strongly agree), suggesting that, on average, centers were
incorporating youth-ownership-related practices into their work with youth. Figures 30 and 31,
however, show that both the majority of site coordinator responses (51.18 percent) and of center
responses (59.67 percent) fell within the agree category. It should be noted that there was a
significant difference from the result in the evaluation report for the 2010–11 school year, where
54 percent of site coordinator respondents fell in the disagree category and 50 percent of center
responses fell in the agree category. This change may indicate that site coordinators and staff,
although with disparate perceptions previously, tend to agree on staff members’ efforts to
develop youth ownership in the program.
Table 16. Statewide Performance on the Youth Ownership Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Youth Ownership—Site Coordinator Survey The extent to which the site coordinator perceives program
staff extending opportunities to youth to develop ownership
in the program.
59.19
(N = 185)
Disagree
(N = 185)
Youth Ownership—Staff Survey The extent to which program staff perceive staff members
extending opportunities to youth to develop ownership in
the program.
59.93
(N = 1030)
Agree
(N = 1030)
Data Source: Site Coordinator and Staff Surveys
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—49
Figure 30. Distribution of Site Coordinator Scale Scores for Survey Questions About
Youth Ownership
Source: Site Coordinator Survey (185 responses)
Figure 31. Distribution of Center Scale Scores for Survey Questions About Youth
Ownership Based on Staff Survey Responses
Source: Staff Survey (1,030 responses from 180 centers)
1.18%
46.47%
51.18%
1.18%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Strongly Disagree Disagree Agree Strongly Agree
Per
cen
t
1.11%
37.02%
59.67%
2.21%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Strongly Disagree Disagree Agree Strongly Agree
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—50
Capacity to Create Interactive and Engaging Settings
Unlike previously described scales, questions related to the capacity to create interactive and
engaging settings ask staff to rate the collective practices of all frontline staff as opposed to
rating their own practice. There is an assumption that staff working in the program have
sufficient interactions with other frontline staff to provide accurate ratings of how well the center
as a whole implements practices that will result in an interactive and engaging setting.
Scale scores for the capacity to create interactive and engaging settings include staff responses to
the following survey question:
PROMPT: Please rate the extent to which you agree or disagree with the following statements
regarding all staff that work with students in this program:
Program staff provides youth the opportunity to engage in group discussion and dialogue
more than placing youth in the role of passive listeners to a lesson or lecture delivered by
staff.
Program staff actively and continuously consults and involves youth.
Program staff provides structured and planned activities explicitly designed to help youth
get to know one another.
Program staff provides opportunities for youth to lead activities.
Program staff provides opportunities for youth to help or mentor other youth in
completing a project or task.
Program staff provides opportunities for the work, achievements, or accomplishments of
youth to be publicly recognized.
Program staff provides ongoing opportunities for youth to reflect on their experiences
(e.g., formal journal writing, informal conversational feedback).
Program staff is effective at finding ways to provide youth with meaningful choices when
delivering activities.
Program staff is effective at providing youth with opportunities to set goals and make
plans within the confines of the program.
Program staff asks for and listens to student opinions about the way things should work
in the program.
As shown in Table 17, the statewide average scale score on the capacity to create interactive and
engaging settings scale fell within the agree response category, suggesting that staff generally
perceive that frontline staff adopt many of the practices listed above. As shown in Figure 32, the
majority (69.41 percent) of centers fell in the agree response category, and 12.35 percent fell in
the disagree response category. This is consistent with prior findings from the evaluation team,
where staff are more confident in their collective ability to create interactive and engaging
settings for youth as compared with their individual practice.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—51
Table 17. Statewide Performance on the Capacity to Create Interactive and Engaging
Settings Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Capacity to Create Interactive and Engaging Settings The extent to which staff members perceive program staff
extending opportunities and providing supports to youth
that result in the creation of an interactive and engaging
setting for participating youth.
60.95
(N = 185)
Agree
(N = 185)
Data Source: Staff Survey
Figure 32. Distribution of Center Scale Scores for Survey Questions About
Staff Capacity to Create Interactive and Engaging Settings Based on Staff Survey
Responses
Source: Staff Survey (185 responses from 180 centers)
Service Delivery Practices
Finally, the service delivery practices scale focuses on how frequently staff report adopting
practices that are likely to foster an interactive and engaging environment for participating youth
(i.e., individual as opposed to collective practice). The list of practices represented on the scale is
in no way meant to be comprehensive but is generally aligned to specific youth development
principles represented on the YPQA.
12.35%
69.41%
18.24%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Disagree Agree Strongly Agree
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—52
Scale scores for service delivery practices include staff responses to the following survey question:
PROMPT: How often are students who are participating in the activities you provide in the
program afforded the following types of opportunities?
Work collaboratively with other students in small groups.
Have the freedom to choose what activities or projects they are going to work on or
participate in.
Work on group projects that take more than one day to complete.
Lead group activities.
Provide feedback on the activities they are participating in during time set aside explicitly
for this purpose.
Participate in activities that are specifically designed to help students get to know one
another.
Make formal presentations to the larger group of students.
As shown in Table 18, the average scale score for staff on the service delivery practices scale fell
within the available occasionally response category (response options included never available,
available occasionally, available regularly, and always available). As Figure 33 shows, 56.35
percent of centers fell in the available occasionally response category.
Compared with staff responses for the 2010–11 school year, the least frequently provided
opportunities shifted from Lead group activities and Make formal presentations to the larger
group of students to Participate in a sequence of sessions where task complexity increases to
build specific skills and Make formal presentations to the larger group of students (see Table
19). Here again, there are certainly opportunities for growth, and it would seem that the YPQI
initiative is conducive to supporting further adoption of these practices in developmentally
appropriate ways.
Table 18. Statewide Performance on the Service Delivery Practices Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Service Delivery Practices—The frequency with which
staff adopt practices that support youth development.
58.7
(N = 1,066)
Available
Occasionally
(N = 1,066)
Data Source: Staff Survey
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—53
Table 19. The Least Frequently Provided Opportunities
Opportunity 2010-2011 School Year 2011-2012 School Year
Make formal presentations to the larger
group of students √ √
Lead group activities √
Participate in a sequence of sessions
where task complexity increases to
build specific skills
√
Data Source: Staff Survey
Figure 33. Distribution of Center Scale Scores for Survey Questions About
Service Delivery Practices Based on Staff Survey Responses
Source: Staff Survey (1066 responses from 180 centers)
Summary of Instructional Context Findings and Recommendations
In many respects, of all the leading indicators, those within the instructional context are
potentially of greatest importance in ensuring high-quality programming due to the fact that
quality at the point of service is how youth experience and benefit from programming. On
average, centers are doing reasonably well in adopting both content- and process-related
practices and approaches associated with the (1) alignment of program activities to identified
program objectives and (2) intentionality of content delivered and support of the provision of
developmentally appropriate settings. However, there is room for growth in each of these areas,
particularly in relation to enhancing intentionality in activity session design and delivery and
providing opportunities for youth ownership.
0.55%
56.35%
38.67%
4.42%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Never Available Available
Occasionally
Available Regularly Always Available
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—54
In order to better target program improvement efforts, more information is needed about the
following areas:
How does performance on leading indicators in the instructional context vary by the
grade level of students served by the program? The concern here is that several of the
process indicators may be more relevant to programs serving youth in secondary grades,
which may warrant exploring different measurement strategies depending on the grade
level of the youth in question.
How does performance on leading indicators in the instructional context vary by center
enrollment in the YPQI process? Here again, many of the items underpinning the
indicators are conceptually related to YPQA, and it is anticipated that centers enrolled in
YPQI will have higher scale scores as a result. Verification of this hypothesis would
enhance the validity of the data resulting in the survey process.
How much variation exists within and across centers in terms of the adoption of high-
quality instructional practice, and how can this information be communicated to centers
in a way to support program improvement efforts without penalizing individual centers
and/or staff?
Each of these questions has implications for the continued development of the leading indicator
system and warrants further exploration in Year 3 evaluation efforts.
Mutually Reinforcing Context
The Mutually Reinforcing Context focuses on relationships between the 21st CCLC program
and contexts external to the program that significantly impact the success of the program.
Community partners, families, and schools play an important role in 21st CCLC programs by
expanding program activities, facilitating program sustainability, and providing important
information about student needs. Three leading indicators are associated with the Mutually
Reinforcing Context: (1) Family Engagement, (2) School Context, and (3) Community Context.
Indicator 3.1: Family Engagement
Engaging families in programming and providing family learning events is an important
component of 21st CCLC programs. Programs may engage families by communicating with
them about center programming and events, collaborating to enhance their child’s educational
success, and providing family literacy and/or social events. Survey questions on the site
coordinator survey measured center approaches to family communication.
Scale scores for family engagement included site coordinator responses to the following survey
questions:
PROMPT: How often do you ...
Send materials about program offerings home to parents/adult family members?
Send information home about how the student is progressing in the program?
Hold events or meetings to which parents/adult family members are invited?
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—55
Have conversations with parents/adult family members over the phone?
Meet with one or more parents/adult family members?
Ask for input from parents/adult family members on what and how activities should be
provided?
Encourage parents/adult family members to participate in center-provided programming
meant to support their acquisition of knowledge or skills?
Encourage parents/adult family members to participate in center-provided programming
with their children?
As shown in Table 20, the average family communication scale score fell within the sometimes
response category (response options were never, sometimes, and frequently), which is indicative
of programs typically communicating with families once or twice a semester. As Figure 34
shows, 73.53 percent of site coordinator respondents fell in the sometimes response category.
The least common family communication strategies included sending information home about
how the student is progressing in the program and asking for input from parents/adult family
members on what and how activities should be provided. The former is not surprising given the
difficulty associated with providing individual progress reports on specific students. However,
the latter is more surprising considering that obtaining feedback from parents/adult family
members is not an overly burdensome or costly task. There might be an opportunity for local
evaluators to assist programs in collecting feedback from parents/adult family members.
Table 20. Statewide Performance on the Family Communication Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Family Communication The frequency with which staff adopt practices that support
communication with parents and adult family members.
59.64
(N = 185)
Sometimes
(N = 185)
Data Source: Site Coordinator Survey
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—56
Figure 34. Summary of Site Coordinator Responses Regarding Family Engagement
Source: Site Coordinator Survey (185 responses)
Indicator 3.2: School Context
This leading indicator is meant to capture the degree to which 21st CCLC staff members align
the design and delivery of programming to the school day and individual student needs. These
practices are particularly important to 21st CCLC program quality, given the explicit goal of
supporting low-performing students’ growth in reading and mathematics. The data reported for
this leading indicator were scored with Rasch scale scores ranging from 0 to 100, where higher
scores are indicative of higher performance/endorsement on a given scale. Four separate scale
scores were calculated for this indicator:
Linkages to the School Day—Site Coordinator Survey: The extent to which the site
coordinator reports taking steps to establish links to the school day and use student data
to inform programming
Linkages to the School Day—Staff Survey: The extent to which program staff report
taking steps to establish links to the school day and use student data to inform
programming
Data Use—Site Coordinator Survey: The extent to which the site coordinator reports
the program using student data to inform programming
Data Use —Staff Survey: The extent to which program staff report taking steps to use
student data to inform programming
It is important to note that the items for linkages to the school day scales on the site coordinator
and staff surveys were quite different. On the site coordinator survey, items were designed to ask
about specific strategies adopted by the program to establish meaningful links to the school day.
11.76%
73.53%
14.71%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Never Sometimes Frequently
Sit
e C
oo
rdin
ato
rs
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—57
Site coordinators were asked to indicate if the strategy described in a given item was (1) a major
strategy, (2) a minor strategy, or (3) not a strategy to support links with the school day. In
contrast, the staff survey asked respondents to indicate their level of agreement with a series of
items regarding their knowledge of school day practices, student academic needs, use of student
data to inform programming, and communication with school day staff to better support the
design and delivery of afterschool programming.
Scale scores for site coordinator responses included the following survey question:
PROMPT: Please indicate whether you receive each of the following, and to what extent you use
it in planning for the activities you provide:
The program has access to individualized education plans, and staff use this information
to plan activities.
The program has access to students’ state assessment scores, and staff use this
information to plan activities.
The program has access to students’ scores on district- or building-level assessments, and
staff use this information to plan activities.
The program has access to students’ grades, and staff use this information to plan
activities.
The program has access to teacher-provided student progress reports/teacher input, and
staff use this information to plan activities.
Scale scores for staff survey responses included the following survey questions:
PROMPT: Please rate the extent to which you agree or disagree with the following statements
regarding linkages to the school day:
On a week-to-week basis, staff know what academic content will be covered during the
school day with the students they work with in the afterschool program.
Staff coordinate the content of the afterschool activities they provide with my students’
school day homework.
Staff know who to contact at their students’ day school if they have a question about their
progress or status.
The activities staff provide in the afterschool program are tied to specific learning goals
that are related to the school day curriculum.
Staff use student assessment data to provide different types of instruction to students
attending their afterschool activities based on their ability level.
Staff monitor students’ academic performance on district- or building-level assessments
across the school year and use this information to inform activities they provide.
Staff help manage a formal three-way communication system that links parents, program,
and day school information.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—58
Staff participate in regular, joint staff meetings for afterschool and regular school day
staff where steps to further establish linkages between the school day and afterschool are
discussed.
Staff meet regularly with school day staff not working in the afterschool program to
review the academic progress of individual students.
Staff participate in parent-teacher conferences to provide information about how
individual students are faring in the afterschool program.
PROMPT: What strategies are used to link the program to the regular school day?
Align programming to school day curriculum and standards.
Help with homework.
Hire regular school day teachers.
Use student assessment and/or grades to inform programming.
Regular face-to-face meetings with school day staff.
Regular electronic communication with school day staff.
Regular electronic communications with principals and other school day administrative staff.
Regular monitoring of students’ academic performance on district- or building-level
assessments across the school year and use of this information to inform activity provision.
Ensure that activities are informed by and meant to support schoolwide improvement
targets related to student performance.
As shown in Table 21, the average site coordinator scale score fell within the minor strategy
response category, indicating that most sites employed only a portion of the listed strategies for
establishing linkages with the school day. As Figure 35a further shows, 53.22 percent of site
coordinator respondents fell within the minor strategy response category, and 31.58 percent fell
within the major strategy category. The least frequently adopted strategy was hiring regular
school day teachers to support links to the school day, and the most common strategy shifted
from align programming to school day curriculum and standards to help with homework. For
staff responses, the average scale score on linkages to the school day fell within the disagree
response category, suggesting that, on average, most staff have an incomplete sense of both
student academic needs and school day curriculum and/or instructions. As shown in Figure 36a,
30.94 percent of centers fell in the disagree response category, and 65.19 percent fell in the
agree category.
The average scale score on data use for both site coordinators and staff fell in the occasionally
use category, indicating the degree to which site coordinators and staff use data to inform
programming is still limited. As shown in Figure 35b and Figure 36b, 72.51 percent of site
coordinator respondents and 54.55 percent of staff responses fell in the occasionally use
category.
It is important to note that no effort was made to control for the staff person’s role in the
program, so staff responsible for the delivery of arts or recreation programming also were asked
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—59
questions related to links to the school day. Finally, responses to items related to the use of
student data to inform programming indicated that these practices were the least common
strategy used by staff to intentionally link programming to the school day. This finding is a
common finding among the evaluations conducted by the evaluation team. Here again, there may
be a role for local evaluators in both gathering and analyzing student data to better support
individual student needs.
Table 21. Statewide Performance on the Linkages to the School Day Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Linkages to the School Day—Site Coordinator Survey The extent to which the site coordinator reports the
program taking steps to establish linkages to the school day
and using student data to inform programming.
60.55
(N = 186)
Minor Strategy
(N = 186)
Linkages to the School Day—Staff Survey The extent to which program staff report taking steps to
establish linkages to the school day and using student data
to inform programming.
58.66
(N = 1,038)
Disagree
(N = 1,038)
Data Use – Site Coordinator Survey The extent to which the site coordinator reports the
program using student data to inform programming.
60.47
(N = 186)
Occasionally Use
(N = 186)
Data Use – Staff Survey The extent to which program staff report taking steps to use
student data to inform programming.
57.94
(N = 915)
Occasionally Use
(N = 915)
Data Source: Site Coordinator and Staff Surveys
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—60
Figure 35a. Summary of Responses Regarding Linkages With the School Day
From the Site Coordinator Survey
Source: Site Coordinator Survey (186 responses)
Figure 35b. Summary of Responses Regarding Data Use From the Site Coordinator
Survey
Source: Site Coordinator Survey (186 responses)
31.58%
53.22%
15.20%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Major Strategy Minor Strategy Not a Strategy
Sit
e C
oo
rdin
ato
rs
14.04%
72.51%
13.45%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
Do Not Receive Occasionally Use Often Use
Sit
e C
oo
rdin
ato
rs
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—61
Figure 36a. Center Classification Based on Staff Members’ Responses
Regarding Linkages to the School Day
Source: Staff Survey (1,038 responses from 180 centers)
Figure 36b. Center Classification Based on Staff Members’ Responses
Regarding Data Use
Source: Staff Survey (915 responses from 180 centers)
2.21%
30.94%
65.19%
1.66%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Strongly Disagree Disagree Agree Strongly Agree
Cen
ters
22.16%
54.55%
23.30%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Do Not Receive Occasionally Use Often Use
Cen
ters
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—62
Indicator 3.3: Community Context
Encouraging partnerships between schools and community organizations is an important
component of the national 21st CCLC programs. Partners are defined as any organization other
than the grantee that actively contributes to a 21st CCLC-funded program to help programs meet
their goals and objectives. Partners may play a variety of roles in supporting a 21st CCLC-
funded program. For example, partners may provide programming and staff, provide physical
space and facilities, and facilitate fundraising efforts. In many instances, partners can play a
critical role in providing activities and services that the grantee lacks expertise or training in
to enhance the variety of learning opportunities available to youth.
From a quality perspective, mutually beneficial partnerships are most effective when staff from
the partner organization work directly with youth and are involved in regular program processes
related to staff orientation, training, evaluation, feedback, and professional development.
The leading indicator for community context is meant to capture the degree to which partners
associated with the center are actively involved in planning, decision making, evaluating, and
supporting program operations. Two separate metrics were calculated to describe aspects of this
indicator:
Partner Involvement: The extent to which partners associated with the center are
actively involved in planning, decision making, evaluating, and supporting the operations
of the afterschool program
Family and Community—YPQA Form B: The extent to which the program adopts
policies and practices supportive of family and community engagement
Partner Involvement
Scale scores for community context include site coordinator responses to the following survey
question:
PROMPT: Do you and representatives from partner agencies involved in afterschool
programming work together to do the following, and if you do, are these things done informally
or formally?
Establish goals and objectives for the program.
Orient new staff to the program.
Provide professional development opportunities to program staff.
Review evaluation results and target areas for improvement.
Develop and evaluate the effectiveness of operational procedures (e.g., recruitment,
scheduling, activity transitions, etc.).
Plan for program sustainability and/or expansion.
As shown in Table 22, the average statewide scale score on the partner involvement scale of the
site coordinator survey fell within the do informally response category (response options
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—63
included did not do, do informally, and do formally). Generally, while centers work with partners
in many of the ways described in the items appearing on this scale, they have a tendency to do so
on an informal basis as opposed to having formal policies and procedures in place to support
partnerships. Figure 37 shows that 18.81 percent of responding site coordinators fell within the
do formally response category, and 59.41 percent fell within the do informally response category.
It is also important to note that only 112 centers had partners that are actively involved in the
provision of programming directly to youth; this represents a little over half of the total centers
with site coordinator survey responses. This may indicate that more time and attention could be
dedicated to finding community partners that can enhance program activities through more direct
involvement in program provision.
Table 22. Statewide Performance on the Linkages to the Partner Involvement Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Partner Involvement The extent to which partners associated with the center are
actively involved in planning, decision making, evaluating,
and supporting the operations of the afterschool program.
58.13
(N = 112)
Do informally
(N = 112)
Data Source: Site Coordinator Survey
Figure 37. Summary of Site Coordinator Responses Regarding Partner Engagement
Source: Site Coordinator Survey (112 responses)
18.81%
59.41%
21.78%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
Do Formally Do Informally Do Not Do
Sit
e C
oo
rdin
ato
rs
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—64
Family and Community
This indicator is meant to capture the barriers to participation and the linkages between program
and families and the community. As shown in Table 23, the statewide scale score on the family
and community fell within the three category. Figure 38 shows that 77.42 percent of staff
respondents fell in this category also. This indicates that a majority of staff reported the policies
and practices of the program promote family and community engagement.
Table 23. Statewide Performance on the Family and Community Scale
Statewide Mean
Range of Scale
Statewide Mean
Fell Within
Family and Community – YPQA Form B
The extent to which the program adopts policies and
practices supportive of family and community engagement.
52.42
(N = 62)
Three
(N = 62)
Data Source: YPQA Form B
Figure 38. Summary of Site Coordinator Responses Regarding Family and Community
Data Source: YPQA Form B
Summary of Findings and Recommendations in Relation to the Community
Context Domain
Of the domains represented in the leading indicator system, the indicators associated with the
Community Context domain are most likely to be influenced by local community contexts.
9.68%
77.42%
12.90%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
One Three Five
Sit
e C
oo
rdin
ato
rs
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—65
For example, Washington is characterized by a relatively large number of grantees that are not
school districts, and as a consequence, the mechanisms for developing effective relationships
with school day staff require both effort and a certain level of trust and rapport, which may have
a bearing on strategies for linking with the school day. In addition, Washington has a relatively
large number of grantees in rural settings (accounting for more than 40 percent of centers active
during the 2011–12 programming period), which can make the process of finding viable
partnerships more complicated, given the limited availability of community partners in rural
settings.
Of the indicators represented in the Community Context domain, it is the opinion of the
evaluation team that the School Context indicator is of greatest import for ensuring high-quality
21st CCLC programming aligned with goals of supporting student growth and development in
reading and mathematics. As with most indicators highlighted thus far in the report, there are
opportunities for growth in relation to establishing links to the school day, particularly in relation
to the use of student data to meet individual student needs. In order to better support grantees in
this regard, it is recommended that steps be taken to modify the leading indicator reports housed
in PPICS to make state assessment data available to grantees so that these data can be analyzed
to better identify student academic needs and inform the design and delivery of programming.
It also may make sense to make this part of the work done by the local evaluator as part of a
revised set of local evaluation guidelines, with guidance around approaches that should be
employed to examine the data and how results should be shared with program management. In
any event, one of the goals of the Year 3 evaluation is to modify the leading indicator reports
based on data associated with the 2012–13 programming period to include this functionality.
Youth Outcomes Leading Indicators
Indicator 4.1: Reading/ELA and Mathematics Performance
This leading indicator is meant to outline the extent to which students attending 21st CCLC
programming regularly (i.e., more than 30 days during the reporting period) moved from one
state proficiency category to another between the 2010–11 and 2011–12 school years in
reading/ELA and mathematics. Similar data are displayed for students attending programming
for fewer than 30 days and students who attended the same schools as 21st CCLC participants
but did not participate in programming. Each cell in Tables 24 and 25 represents the percentage
of students in a given group (i.e., regular attendee, non-regular attendee, or nonparticipant) who
scored in a particular proficiency category in 2010–11 that also scored in a particular proficiency
category in 2011–12. It is important that these data are descriptive in nature and cannot be used
to assess the program's causal impact on student outcomes in reading/ELA and mathematics.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—66
Table 24. Mathematics Performance
2010–
11
2011–12
Level 1 –
Well Below
Standard
Level 2 –
Below
Standard
Level 3 –
Met
Standard
Level 4 –
Above
Standard
Center Data,
Regular
Attendees
Level 4 –
Above
Standard
1.52% 1.52%
Level 3 –
Met Standard 1.52% 3.03% 9.09%
Level 2 –
Below
Standard
10.61% 24.24% 7.58% 1.52%
Level 1 –
Well Below
Standard
25.76% 13.64%
Center Data,
Non-Regular
Attendees
Level 4 –
Above
Standard
Level 3 –
Met Standard 10%
Level 2 –
Below
Standard
20% 10% 10%
Level 1 –
Well Below
Standard
20% 20% 10%
Feeder School
Data
Level 4 –
Above
Standard
0.29% 2.92% 11.99%
Level 3 –
Met Standard 1.46% 6.14% 22.22% 9.06%
Level 2 –
Below
Standard
7.60% 6.73% 6.73% 0.29%
Level 1 –
Well Below
Standard
19.59% 3.51% 1.46%
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—67
Table 25. Reading/ELA Performance
2010–
11
2011–12
Level 1 –
Well Below
Standard
Level 2 –
Below
Standard
Level 3 –
Met
Standard
Level 4 –
Above
Standard
Center Data,
Regular
Attendees
Level 4 –
Above
Standard
1.61%
Level 3 –
Met Standard 9.68% 9.68% 3.23%
Level 2 –
Below
Standard
8.06% 32.26% 12.90% 4.84%
Level 1 –
Well Below
Standard
8.06% 8.06% 1.61%
Center Data,
Non-Regular
Attendees
Level 4 –
Above
Standard
Level 3 –
Met Standard 12.50%
Level 2 –
Below
Standard
25% 25% 25%
Level 1 –
Well Below
Standard
12.50%
Feeder School
Data
Level 4 –
Above
Standard
1.20% 8.71% 16.22%
Level 3 –
Met Standard 0.60% 4.80% 16.52% 15.92%
Level 2 –
Below
Standard
2.70% 10.51% 6.01% 2.40%
Level 1 –
Well Below
Standard
12.01% 2.10%
0.30%
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—68
Determining Program Improvement Priorities From the
Leading Indicator System
One of the goals of the leading indicator system is to help OSPI make a determination regarding
where efforts should be invested to support programs in the adoption of quality afterschool
practices. For each scale represented in the leading indicator system, a portion of that scale
indicates that a quality approach or practice is largely not being adopted by the center in
question. In Table 26, each of the indicators and related scales are listed along with the portion of
the scale that indicates that a given practice is largely absent from the center in question and the
number and percentage of centers that fall within these ranges. As shown in Table 26, there is no
scale in which more than 50 percent of centers fell within a range indicating that the quality
practice was largely absent, in comparison with the result in the 2010–11 evaluation where there
were two such scales: the youth ownership scale from the site coordinator survey and the
linkages to the school day scale from the staff survey. This change may indicate improvement in
the quality practice of the two indicators.
In addition, it should be noted that there were differences in terms of scales where over 20 percent
of centers reported significant lack of quality practice between 2010–11 and 2011–12
programming periods. As indicated in Table 27, more efforts are still needed for the development
of program climate, youth ownership, and linkages to the school day. In addition, it may be
worthwhile to consider how to promote staff’s data use and partner involvement in program
support.
In terms of climate, our sense is that a lack of time for session planning and preparation is a
substantive issue in these centers, and that it may be worthwhile for OSPI to consider what
message it wants to send to grantees regarding the importance of creating space for lesson planning
and session preparation and how this message could be sent, both in terms of how RFPs are crafted
and the types of professional development and tools provided to project directors and center
coordinators.
As already noted, the linkages to the school day scale was calculated based on all staff members
responding to the survey regardless of staff roles. Staff role relative to providing activities with
an explicit academic focus is being tracked in a more intentional way as part of the survey data
collection process for 2012–13, giving the evaluation team a way to examine adoption of these
practices more thoroughly, based on the staff member’s role in the program. As a result, it is our
sense that it would be premature to draw overly definitive conclusions about staff practice in this
area.
In terms of the youth ownership scale, more definitive conclusions can be reached about the
percentage of centers where these practices are largely absent, and findings from the staff survey
reinforce these results as well as those shown in Table 26. The bigger question here is how to
cultivate center adoption of these practices in a developmentally appropriate fashion that takes
into consideration the diversity of grade levels served by 21st CCLC programs in Washington.
This is a topic that warrants additional consideration during future years of the evaluation. In
terms of data use, our sense is that it may be worthwhile to consider promoting the program’s
access to individual education plans and students’ scores on district and state level assessments,
and engaging staff in using this information to plan activities.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—69
Table 26. Leading Indicator Scales by Number and Percentage of Centers Where Quality
Practices Were Largely Absent
Domain/Scale
Rating Options
Indicating Practice
Not Present
N
Centers
%
Centers
Organization Context
Program Climate The extent to which program staff report that a
supportive and collaborative climate exists within the
program
Disagree, Strongly
Disagree 39 21.67%
Internal Communication—Site Coordinator
Survey The frequency with which the site coordinator engages in
practices with program staff that support internal
communication and collaboration
Never 4 2.35%
Internal Communication—Staff Survey The frequency with which the staff engages in
practices with other program staff that support internal
communication and collaboration
Never 6 3.31%
Leadership and Management—YPQA Form B
The extent to which the program is engaging in
practices that ensure staff are well positioned to
create developmentally appropriate settings for youth
and that processes are in place to support program
improvement efforts
Five 6 9.68%
Instructional Context
Program Objectives—The degree to which site
coordinators’ top three objectives align with proportion
of time spent on corresponding activities
Alignment not present 19 10%
Intentionality in Program Design—Site
Coordinator Survey The frequency with which staff engage in practices
that indicate intentionality in activity and session
design among staff responsible for the delivery of
activities meant to support student growth and
development.
Rarely 14 8.24%
Intentionality in Program Design—Staff Survey The frequency with which staff engage in practices
that indicate intentionality in activity and session
design among staff responsible for the delivery of
activities meant to support student growth and
development.
Rarely 4 2.35%
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—70
Domain/Scale
Rating Options
Indicating Practice
Not Present
N
Centers
%
Centers
Point of Service Quality—YPQA Form A
The extent to which program staff provide supports
and opportunities to create safe, supportive,
interactive, and engaging settings for participating
youth (total YPQA score)
Still Room for
Improvement 14 19.44%
Safe Environment 0 0
Supportive Environment Still Room for
Improvement 8 11.11%
Interaction Functioning Near
Optimal 12 16.67%
Engagement Functioning Near
Optimal 7 9.72%
Youth-Centered Policies & Practices—YPQA
Form B
The extent to which the program adopts youth-
centered policies and practices conducive to a
supportive learning environment
One 6 9.68%
Youth Ownership—Site Coordinator Survey The extent to which the site coordinator perceives
program staff extending opportunities to youth to
develop ownership in the program
Disagree, Strongly
Disagree 81 47.65%
Youth Ownership—Staff Survey The extent to which program staff perceive staff
members extending opportunities to youth to develop
ownership in the program
Disagree, Strongly
Disagree 69 38.13%
Capacity to Create Interactive and Engaging
Settings—The extent to which staff members perceive
program staff extending opportunities and providing
supports to youth that result in the creation of an
interactive and engaging setting for participating youth
Disagree 21 12.35%
Service Delivery Practices—The frequency with
which staff adopt practices that support youth
development
Never 1 0.55%
Mutually Reinforcing Context
Family Communication—The frequency with which
staff adopt practices that support communication with
parents and adult family members
Never 20 11.76%
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—71
Domain/Scale
Rating Options
Indicating Practice
Not Present
N
Centers
%
Centers
Linkages to the School Day—Site Coordinator
Survey The extent to which the site coordinator reports the
program taking steps to establish linkages to the
school day and using student data to inform
programming
Not a strategy 26 15.2%
Linkages to the School Day—Staff Survey The extent to which program staff report taking steps
to establish linkages to the school day and using
student data to inform programming
Disagree, Strongly
Disagree 60 33.15%
Data Use—Site Coordinator Survey
The extent to which the site coordinator reports the
program using student data to inform programming
Do Not Receive 24 14.04%
Data Use—Staff Survey
The extent to which program staff report taking steps
to use student data to inform programming
Do Not Receive 39 22.16%
Partner Involvement The extent to which partners associated with the center
are actively involved in planning, decision making,
evaluating, and supporting the operations of the
afterschool program
Do not do 22 21.78%
Family and Community – Form B PQA
The extent to which the program adopts policies and
practices supportive of family and community
engagement
One 6 9.68%
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—72
Table 27. Scales Where More Than 20 Percent (Including More Than 50 Percent) of
Centers Report Lack of Quality Practice
Scale 2010–11 School Year
(% Centers)
2011–12 School Year
(% Centers)
Program climate √
(27%)
√
(22%)
Capacity to create interactive and
engaging settings
√
(26%)
Program objectives √
(22%)
Youth ownership—site coordinator √
(56%)
√
(48%)
Youth ownership—staff survey √
(48%)
√
(38%)
Linkages to the school day—staff survey √
(52%)
√
(33%)
Data use—staff survey √
(22%)
Partner involvement √
(22%)
Data Source: Staff Survey
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—73
Chapter 5: Assessing 21st CCLC Program Outcomes
Within-Program and Impact Analyses
Another primary objective of the statewide evaluation was to understand the relationship
between participation in 21st CCLC-funded programs and student academic behaviors and
outcomes. Employing program participation and outcome data associated with the 2011–12
programming period, two analytic approaches were used:
Within-program analyses. The within-program analyses examined the relationship
between student outcomes and several student and program characteristics, with a
particular emphasis on exploring the relationship between leading indicator status and
youth outcomes. The analyses are correlational in nature, meaning that inferences about
causation or directionality cannot be made. Other factors that were not included in the
analyses may play a role in the reported findings.
Impact analyses. The impact analyses were based on a rigorous quasi-experimental
design that compared academic outcomes of 21st CCLC program participants with
matched nonparticipating students using a propensity score matching approach.
Meaningful conclusions may be drawn from the impact analysis regarding the impact of
Washington 21st CCLC program participation on student outcomes.
To determine student- and center-level characteristics related to the student outcomes under
consideration, the evaluation team employed a series of hierarchical linear models (HLMs) to
test for statistically significant relationships between student and center characteristics and a
variety of youth outcomes. Findings from these analyses are described in the following section.
Within-Program Analyses
Given the time and effort involved in both collecting and analyzing the data needed to support
population of the leading indicators, it seemed appropriate to explore if a center’s status on a
given set of leading indicators was related to (a) the degree to which youth participated in the
21st CCLC program during the 2011–12 project period and (b) performance on school-related
outcomes. In order to both simplify these analyses and triangulate data from multiple leading
indicators related to a given domain of center performance, hierarchical cluster analysis was used
to classify centers into one of three primary subgroups based on the center’s performance across
indicators in that domain:
When all indicators suggest above average performance in relation to a given
quality domain. These are centers that may warrant further examination to learn more
about the strategies that support effective implementation of quality practices.
It would also be expected that the likelihood that such centers would have a positive
impact on student outcomes would be greater.
When all indicators suggest below average performance in relation to a given
quality domain. These are centers that could especially benefit from additional services
and supports to enhance the quality of program implementation. Similar to the
information presented in Table 26, knowing how many centers fall within this category
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—74
across the various quality domains could prove useful to OSPI as it structures and
prioritizes its technical assistance and training agenda.
Mismatches in indicators in relation to a given quality element. These are centers in
which there is divergence in the indicators of implementation within a given domain.
These mismatches may suggest a lack of communication and shared vision and
understanding among key actors within the program. In these centers, consideration could
be given to achieving a shared vision and understanding of the goals, planning
requirements, implementation characteristics (e.g., high-level planning and management
and day-to-day tasks), program improvement strategies, challenges, and data/outcomes
associated with effective implementation of 21st CCLC programming.
Hierarchical cluster analyses were performed in relation to three leading indicator domains:
(1) Organizational Context, (2) Instructional Context-Content, and (3) Instructional Context-
Process. The next section summarizes results from each cluster analysis, and then steps are taken
to highlight findings that summarize the relationship between cluster membership status and
program attendance and school-related outcomes.
Organizational Context Clusters
Center performance on four indicators associated with the Organizational Context domain was
used to form clusters in this domain.2
Training Participation
Program Climate
Site Coordinator-Reported Internal Communication
Staff-Reported Internal Communication
As shown in in Figure 39, application of hierarchical cluster analysis techniques yielded the
following three quality profiles:
1. Site Coordinator Internal Communication Score Especially Above Average. A total
of 49 centers (29 percent of the total with complete indicator data) were assigned to this
cluster where scores on three of the four leading indicators under consideration were
above average, particularly the site coordinator’s score on the internal communication
scale. Centers in this cluster would be considered to have a higher degree of
implementation on strategies and practices that support staff development and internal
communication and collaboration among program staff.
2. All Indicators Below Average. A total of 92 centers (54 percent of the total with
complete indicator data) were assigned to this cluster where scores on all four leading
indicators under consideration were below average. Centers in this cluster would be
considered to have a lower degree of implementation on strategies and practices that
2 It is important to note that indicators predicated on YPQA data were not included in any of the hierarchical cluster
analyses detailed in this section of the report. Hierarchical cluster analysis requires complete data across all fields
included in the analysis. Including YPQA data would have greatly reduced the number of centers that could be
included in these analyses because only a portion of the 21st CCLC population of centers participated in the YPQI
process during 2011–12.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—75
support staff development and internal communication and collaboration among program
staff.
3. Staff Climate and Internal Communication Scores Above Average. A total of 28
centers (17 percent of the total with complete indicator data) were assigned to this cluster
where scores on indicators based on data collected from activity leaders working in 21st
CCLC programs were especially above average, while the average site coordinator score
on the internal communication scale was below average. This disconnect between staff
and site coordinator scores among centers in this cluster is of particular interest, possibly
demonstrating more staff-to-staff interaction and communication than similar processes
occurring between the site coordinator and staff.
Figure 39. Clusters Related to Strategies and Practices That Support a High-Quality
Organizational Context
Source: 169 centers with data on leading indicators related to Organizational Context
Instructional Context-Content Clusters
Center performance on six indicators associated with the Instructional Context-Content domain
was used to form clusters in this domain:
Site Coordinator-Reported Intentionality in Program Design
Staff-Reported Intentionality in Program Design
Site Coordinator-Reported Linkages to the School Day
Staff-Reported Linkages to the School Day
Site Coordinator-Reported Use of Student Data to Inform Programming
-1.5
-1
-0.5
0
0.5
1
1.5
SC comm above
average (29%)
All below average
(54%)
Staff climate and
comm above
average (17%)
Mea
n S
tan
da
rdiz
ed S
core
Mean Training Attendance
Mean Climate Score
Mean SC Internal Comm Score
Mean Staff Internal Comm Score
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—76
Staff-Reported Use of Student Data to Inform Programming
As shown in in Figure 40, application of hierarchical cluster analysis techniques yielded three
quality profiles:
1. Site Coordinator Scores Above Average. A total of 75 centers (46 percent of the total
with complete indicator data) were assigned to this cluster where scores on three of the
six leading indicators under consideration were noticeably above average, all of which
pertained to the site coordinator’s perception on indicators related to the Instructional
Context-Content domain. This may be an indication that ownership of practices
associated with the alignment of programming to school-related considerations may
reside primarily with site coordinators in this cluster, as compared with a more shared
approach where frontline delivery staff share significant responsibility in engaging in
such practices and approaches to support the content delivery objectives associated with
the center’s program.
2. All Indicators Above Average. A total of 21 centers (13 percent of the total with
complete indicator data) were assigned to this cluster where scores on all six leading
indicators under consideration were above average. Generally, centers in this cluster
would be considered to have a higher degree of implementation on strategies and
practices that support the embedding of academic content into program activities aligned
with the school day.
3. All Indicators Below Average. A total of 67 centers (41 percent of the total with
complete indicator data) were assigned to this cluster where scores on all six leading
indicators under consideration were below average. This was particularly the case in
relation to indicators based on site coordinators’ perceptions of linkages with the school
day and use of school-related data to inform the design and delivery of programs.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—77
Figure 40. Clusters Related to Strategies and Practices That Support a High-Quality
Instructional Context-Content
Source: 163 centers with data on leading indicators related to Instructional Context-Content
Instructional Context-Process Clusters
Center performance on four indicators associated with the Instructional Context-Process domain
was used to form four clusters in this domain:
Site Coordinator-Reported Opportunities for Youth Ownership
Staff-Reported Opportunities for Youth Ownership
Site Coordinator-Reported Efficacy of Staff in Creating Interactive and Engaging
Settings
Staff-Reported Use of Youth Development Practices
As shown in in Figure 41, application of hierarchical cluster analysis techniques yielded three
quality profiles:
1. Site Coordinator Scores Above Average. A total of 100 centers (59 percent of the total
with complete indicator data) were assigned to this cluster where scores on two of the
four leading indicators under consideration were noticeably above average, all of which
pertained to the site coordinator’s perception on indictors related to the Instructional
Context-Process domain. This may be an indication that ownership of practices
associated with the provision of youth ownership opportunities may reside at the program
level through opportunities such as youth program governance as opposed to being a core
aspect of activity delivery at the point of service. Site coordinators in this cluster also
believed their staff members were engaging in practices that would create an interactive
and engaging environment for participating youth.
-1.5
-1
-0.5
0
0.5
1
1.5
SC above average
(46%)
All above average
(13%)
All below average
(41%)
Mea
n S
tam
da
rdiz
ed S
core
Mean SC design score
Mean staff design score
Mean SC linkages score
Mean staff linkages score
Mean SC data use score
Mean staff data use score
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—78
2. Staff Scores Above Average. A total of 43 centers (25 percent of the total with complete
indicator data) were assigned to this cluster where scores on two of the four leading
indicators under consideration were above average, although in this case the indicators
well above average were associated with the staff survey. This result is somewhat of a
curious one, suggesting that staff members are either overestimating the provision of such
opportunities to youth in their own practice or that site coordinators in such centers are
largely unaware of what their staff are doing in regards to engaging in practices and
approaches designed to support youth development and ownership in the program.
3. All Indicators Below Average. A total of 26 centers (15 percent of the total with
complete indicator data) were assigned to this cluster where scores on all four leading
indicators under consideration were below average. This was particularly the case in
relation to indicators based on staff perceptions of the provision of opportunities for
youth development and ownership.
Figure 41. Clusters Related to Strategies and Practices That Support a High-Quality
Instructional Context-Process
Source: 169 centers with data on leading indicators related to Instructional Context-Process
Primary Hypothesis Tested Through Within-Program Analyses
The within-program analyses highlighted in the sections that follow were primarily oriented at
exploring the veracity of one primary hypothesis related to the assignment of centers to different
clusters constructed from the leading indicators by domain:
-1.5
-1
-0.5
0
0.5
1
1.5
SC above average
(59%)
Staff above average
(25%)
All below average
(15%)
Mea
n S
tan
da
rdiz
ed S
core
Mean SC onwnership scale
Mean staff ownership scale
Mean SC efficacy scale
Mean staff YD practices scale
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—79
It was hypothesized that there would be a negative correlation between center
membership in the All Indicators Below Average cluster for each domain and youth
program attendance and outcomes.
As noted in the prior sections, each domain was found to have one cluster where centers were
below average on each of the indicators represented in that domain, although as shown in Figures
39–40, the percentage of centers falling in the All Indicators Below Average cluster varied
significantly by domain:
Organizational Context: 54 percent of centers fell in the All Indicators Below Average
cluster.
Instructional Context-Content: 41 percent of center fell in the All Indicators Below
Average cluster.
Instructional Context-Process: 15 percent of center fell in the All Indicators Below
Average cluster.
A decision was made to focus on centers classified in the All Indicators Below Average cluster
based on an assumption that issues of social desirability in response patterns would be less of an
issue in relation to scores associated with centers in these clusters, and consequentially, estimates
derived from site coordinator and staff surveys may be more indicative of the actual level of
functioning at these centers. In this sense, it was expected that there would be a negative
relationship between membership in an All Indicators Below Average cluster and the attendance
school-related outcomes examined.
Summary of Within-Program Analysis Variables
In addition to whether or not a given center fell in an All Indicators Below Average cluster, the
multilevel models underpinning the within-program analyses explored associations between a
variety of student- and center-level characteristics and student outcomes for 21st CCLC
participants. Student outcomes included the following:
Youth participation in the 21st CCLC program during the 2011–12 project period (all
grades)
2012 Measurements of Student Progress (MSP) reading assessment performance
(Grades 4–8)
2012 MSP mathematics assessment performance (Grades 4–8)
2012 High School Proficiency Exam (HSPE) reading assessment performance (Grade 10)
Cumulative grade point average (GPA) (Grades 9–12)
Percentage of credits earned relative to credits attempted (Grades 9–12)
Number of unexcused absences for the 2011–12 school year (all grades)
In addition to the aforementioned youth outcomes, a variety of other student-level characteristics
were included in the within-program models:
Youth grade level
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—80
Status as a racial minority
Hispanic ethnicity
Gender
Eligibility for free or reduced-price lunch
Special education status
Limited English proficiency status
A summary of youth characteristics across each of these areas is displayed in Table 28. As
shown, most youth included in the within-program analyses were in Grades 4–8 (75 percent), the
majority were identified as racial minorities (71 percent), there were slightly more males than
females (51versus 49 percent, respectively), about four fifths (83 percent) qualified for free or
reduced-price lunch, 20 percent were designated as having limited proficiency in English, and 14
percent were receiving special education services.
Table 28. Summary Statistics: Student Characteristics
Proportion of 2011–12
21st CCLC Participants
Grade Level (n = 14,925)
3 0.099
4 0.105
5 0.104
6 0.196
7 0.192
8 0.152
9 0.048
10 0.059
11 0.030
12 0.016
Minority Status (n = 14,900)
Minority 0.707
Nonminority 0.293
Hispanic Status (n = 14,900)
Yes 0.398
No 0.602
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—81
Proportion of 2011–12
21st CCLC Participants
Gender (n =14,925)
Male 0.514
Female 0.486
Free or Reduced-Price Lunch Status (n = 14,871)
Eligible 0.826
Not eligible 0.174
Special Education Status (n = 14,871)
Yes 0.140
No 0.860
Limited English Proficiency (n = 14,871)
Yes 0.200
No 0.800
As shown in Table 29, in addition to whether or not a given center fell in an All Indicators Below
Average cluster, other center-level characteristics included in the within-program models
included the following:
Whether or not the center was associated with a school-based grantee (45 percent)
Whether or not the center was staffed mostly by school day teachers (45 percent)
Whether or not the site coordinator had at least a bachelor’s degree (77 percent)
These center-level characteristics were found to be significant in within-program models run in
relation to data associated with the 2010–11 program period and, therefore, were considered
viable candidates for inclusion in the analyses undertaken in relation to 2011–12.
It is also important to note that HLM requires complete data at the center level for all variables
included in the model. In this regard, although cluster assignment information was available for
169 centers for the Organizational Context and the Instructional Context-Process domain, only
163 centers had data in relation to the Instructional Context-Content domain. As a result, 163
centers were included in the within-program analyses for 2011–12.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—82
Table 29. Summary Statistics: Center Characteristics
Proportion of 2011–12
21st CCLC Centers
Grantee Type (n = 163)
School-based 0.454
Non-school-based 0.546
Staffing Cluster (n = 163)
Mostly teachers 0.454
All other staffing clusters 0.546
Site Coordinator has bachelor degree or above (163)
Yes 0.773
No 0.227
Leading Indicator Cluster: Organizational Context Below Average (n = 163)
Yes 0.467
No 0.534
Leading Indicator Cluster: Instructional Context-Content Below Average (n = 163)
Yes 0.411
No 0.589
Leading Indicator Cluster: Instructional Context-Process Below Average (n = 163)
Yes 0.147
No 0.853
Within-Program Analysis Results and 21st CCLC Program Attendance
Results outlining the relationship between center and youth characteristics and the number of
days attending 21st CCLC programming during the 2011–12 project period are outlined in Table
30.
Again, the primary goal of the within-program analysis was to explore if a relationship could be
found between membership in a cluster indicating below average levels of implementation on the
practices associated with a particular leading indicator domain and youth outcomes. As shown in
Table 30, a significant, negative relationship was found between center membership in the
Organizational Context Below Average cluster and youth attendance in 21st CCLC. This
relationship is consistent with our hypothesis, indicating that 21st CCLC attendance was lower in
centers where adoption of practices associated with the Organizational Context domain was
below average.
Contrary to our hypothesis, however, center membership in the Instructional Context Process
Below Average cluster was found to be significantly and positively related to 21st CCLC
attendance, indicating a higher level of attendance in centers falling into this cluster. Although
initially puzzling, a further examination of the within-program analysis data set demonstrated
that centers falling in the Instructional Context Process Below Average cluster were much more
likely to be centers that served elementary students only (p < .01). It is a well-established finding
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—83
that the total number of days of attendance in 21st CCLC programming is substantially higher in
elementary programs compared with programs serving middle or high school students. In fact,
this finding is also present in Table 30: Middle and high school youth were found to have
significantly lower levels of attendance in 21st CCLC compared with elementary students. It is
also known that elementary programs are less apt to score as well on the YPQA, which contains
many of the youth-development-related practices represented in the Instructional Context
Process Below Average cluster. It is possible that these core grade-level differences associated
with membership in the Instructional Context Process Below Average cluster are associated with
the positive relationship observed in Table 30.
Other significant findings outlined in Table 30 include the following:
Grant School-Based (Negative Relationship). A negative relationship was found to
exist between a center’s status as being associated with a school-based grantee and youth
attendance in the 21st CCLC program. In this sense, 21st CCLC attendance was found to
be lower in centers run by school-based grantees.
Continuous Years of Enrollment (Positive Relationship). Youth enrolled in 21st
CCLC for two or more years were more likely to have higher program attendance in
2011–12.
Table 30. Model Results: Program Attendance Outcome With
Leading Indicator Predictors
Predictors 21st CCLC Program Attendance
Grant school-based -0.128**
(0.054)
Mostly teachers 0.012
(0.054)
Academic site coordinator has B.A.+ -0.006
(0.062)
Organizational Context below average -0.181***
(0.054)
Instructional Context-Content below average -0.036
(0.053)
Instructional Context-Process below average 0.127*
(0.077)
Slopes
Continuous years 0.192***
(0.010)
Middle school student -0.309***
(0.027)
High school student -0.312***
(0.055)
Free or reduced-price lunch eligible -0.005
(0.013)
Special Education 0.025*
(0.013)
Note. Standard errors are reported in parentheses; *** sig. at 0.01; ** sig. at 0.05;
* sig. at 0.10.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—84
Within-Program Analysis Results and Unexcused Absences
Results outlining the relationship between center and youth characteristics and the number of
unexcused school-day absences during the 2011–12 school year are outlined in Table 31. Given
that unexcused school-day absences are a negative outcome, it was expected that we would see a
positive relationship between membership in a cluster defined by below-average performance on
the leading indicators and the number of unexcused absences accumulated by program youth.
As shown in Table 31, this was found to the case in relation to membership in the Instructional
Context-Content Below Average cluster. In this case, a significant positive relationship was
found between membership in the Instructional Context-Content Below Average cluster and the
number of unexcused school-day absences accumulated by participating youth. Although
consistent with our overall expectation that membership in a below-average cluster would be
associated with less desirable youth outcomes, we considered it more likely that membership in
the Instructional Context Process Below Average cluster would be more apt to be associated with
higher school-day unexcused absences because the youth development aspects of the
programming would help draw and retain students in the 21st CCLC program and by proxy
support school-day attendance. It may be the case that a focus on content also may be associated
with more explicit targeting of youth for participation in programming, which may mean more
communication with parents and guardians, which in turn may facilitate both program and
school-day attendance. At this point, such possible explanations are mere speculation.
In terms of other findings outlined in Table 31, a number of youth characteristics were associated
with higher rates of unexcused absences, including (a) high school students, (b) youth eligible for
free or reduced-price lunch, (c) special education students, and (d) minority youth. Youth
characteristics negatively associated with unexcused absences included (a) Hispanic youth and
(b) students with higher levels of attendance in 21st CCLC. The latter result will be examined
more closely in the impact analysis section of this report where results for 21st CCLC
participants versus non-participating youth attending the same schools are examined.
Finally, youth associated with centers associated with school-based grantees also had higher
levels of unexcused absences than youth attending centers run by non-school-based grantees.
Table 31. Model Results: Unexcused Absences Outcome With
Leading Indicator Predictors
Predictors Number of Unexcused Absences
Grant school-based 0.689**
(0.280)
Mostly teachers -0.065
(0.284)
Academic site coordinator has B.A.+ -0.223
(0.323)
Organizational Context below average 0.209
(0.283)
Instructional Context-Content below average 0.488*
(0.279)
Instructional Context-Process below average -0.258
(0.409)
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—85
Slopes
SY days -0.0060***
(0.001)
Continuous years 0.004
(0.032)
Middle school student -0.113
(0.097)
High school student 0.667***
(0.145)
Free or reduced-price lunch eligible 0.540***
(0.047)
Special education 0.220***
(0.040)
Limited-English-proficient status -0.019
(0.039)
Gender (1 = male) 0.037
(0.029)
Hispanic -0.095**
(0.043)
Minority 0.187***
(0.049)
Notes. Standard errors are reported in parentheses; *** sig. at 0.01; ** sig. at 0.05; * sig. at 0.10.
Within-Program Analysis Results and Academic-Performance-Related Outcomes
Results outlining the relationship between center and youth characteristics and academic
performance-related outcomes associated with the 2011–12 school year are outlined in Table 32.
Outcomes related to academic performance include performance on state assessments in reading
and mathematics (primarily MSP, but also HSPE for Grade 10 reading), cumulative GPA, and
percentage of credits earned.
As shown in Table 32, the only below-average cluster predicated on leading indicator data
negatively associated with the academic-performance-related outcomes examined as part of the
within-program analyses was membership in the Instructional Context Content Below Average
cluster and reading scores. In this sense, students attending centers characterized by below-
average performance on the leading indicators related to the Instructional Context-Content
domain scored significantly lower on reading state assessments taken during the 2011–12 school
year.
The only other result related to cluster membership and academic-performance-related outcomes
pertains to a positive relationship between membership in the Organizational Context Below
Average cluster and both cumulative GPA and credits earned. Because both of these outcomes
pertain to high school students specifically, our sense is that the nature of this unexpected
relationship somehow relates to some element of the operation of high school programs.
In terms of other outcomes, across all academic performance-related outcomes, a significant
positive relationship was found to exist between days of participation in 21st CCLC
programming and academic-performance-related outcomes. The more youth attended
programming, the better they were found to perform academically. In addition, youth attending
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—86
the program for two years or more also were shown to score better on the mathematics portion of
the state assessments.
Yet again, youth associated with centers associated with school-based grantees were found to
demonstrate lower levels of performance on both mathematics and reading assessments, and
youth attending centers staffed mostly by school-day teachers were found to perform better on
mathematics state assessments.
Finally, a host of different student characteristics were found to be related to academic-
performance-related outcomes. Youth who were eligible for free or reduced-price lunch, were
receiving special education services, were limited English proficient, were male, and were
minorities were found to perform lower on one or more academic-performance-related outcomes.
Table 32. Model Results: Achievement Outcomes With Leading Indicator Predictors
Predictors Mathematics Reading/
Language Arts
Cumulative
GPA
Percent of Credits
Earned
Grant school-based -0.074**
(0.033)
-0.046*
(0.026)
-0.181
(0.167)
-0.055
(0.034)
Mostly teachers 0.062*
(0.033)
-0.013
(0.025)
-0.034
(0.192)
-0.025
(0.038)
Academic Site Coordinator has
B.A.+
0.009
(0.038)
0.005
(0.029)
0.017
(0.196)
0.025
(0.039)
Organizational Context below
average
0.025
(0.033)
0.005
(0.025)
0.295*
(0.177)
0.073**
(0.034)
Instructional Context-Content
below average
-0.029
(0.033)
-0.043*
(0.025)
0.022
(0.192)
0.002
(0.038)
Instructional Context-Process
below average
0.072
(0.047)
0.047
(0.036)
-0.572
(0.357)
-0.052
(0.069)
Slopes
2011 standardized score 0.728***
(0.007)
0.722***
(0.007) - -
SY Days 0.001***
(0.0002)
0.001***
(0.0002)
0.004***
(0.001)
0.001***
(0.0002)
Continuous years 0.034**
(0.013)
0.017
(0.014)
-0.041
(0.041)
0.001
(0.010)
Middle school student 0.052*
(0.028)
0.056**
(0.027) - -
High school student - -0.147
(0.269) - -
Free or reduced-price lunch
eligible
-0.081***
(0.017)
-0.082***
(0.017)
-0.257***
(0.054)
-0.056***
(0.013)
Special education -0.252***
(0.018)
-0.287***
(0.019)
-0.151**
(0.063)
0.013
(0.015)
Limited-English-proficient status -0.128***
(0.017)
-0.157***
(0.018)
0.038
(0.053)
0.015
(0.013)
Gender (1 = male) 0.008
(0.012)
-0.035***
(0.012)
-0.279***
(0.038)
-0.051***
(0.009)
Hispanic -0.031
(0.020)
0.004
(0.020)
0.073
(0.054)
0.020
(0.013)
Minority -0.010
(0.020)
-0.034*
(0.020)
-0.146***
(0.056)
-0.025*
(0.013)
Notes. Standard errors are reported in parentheses; *** sig. at 0.01; ** sig. at 0.05; * sig. at 0.10.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—87
Summary of Within-Program Analyses Findings
The primary goal of the within-program analyses was to explore if there was evidence to support
the hypothesis that there would be a negative correlation between center membership in the all
indicators below average cluster for each domain and youth program attendance and outcomes.
The most consistent evidence to support this hypothesis was found in relation to center
membership in the Instructional Context-Content Below Average cluster where the hypothesized
relationship was found to exist in relation to the number of unexcused absences and performance
in reading state assessments.
In addition, membership in the Organizational Context Below Average cluster was also found to
be negatively associated with youth attendance in the program as anticipated, but positively
associated with cumulative GPA and credits earned.
Similar unexpected findings were found in relation to membership in the Instructional Context
Process Below Average and 21st CCLC program attendance where membership in this cluster
was associated with higher levels of program attendance, although it hypothesized that this result
may be driven by the high number of centers serving elementary only represented in this cluster.
Generally, the evidence supporting our hypothesis on the relationship between cluster
membership based on leading indicator performance and youth outcomes was not found to be
consistent or pervasive across all outcomes examined. The most consistent findings were in
relation to the Instructional Context Content Below Average cluster, with similar analyses
conducted as part of the statewide 21st CCLC evaluation in New Jersey demonstrating a similar
connection between content and youth outcomes. Given the percentage of centers falling in the
below-average cluster (41 percent), there are also opportunities to support further growth in
performance on these indicators through the expansion of training and technical assistance
efforts.
Also worthy of note is the finding that a significant, positive relationship was found between the
number of days of 21st CCLC participation and each of the academic-performance-related
outcomes examined. This is the type of relationship one would like to see if one were looking for
evidence that participation in 21st CCLC may be having a positive impact on such outcomes.
Finally, it is important to keep in mind the domain of within-program analyses conducted here
are correlational and descriptive in nature and do not permit causal inferences. For example, the
within-program findings cannot answer the question on whether more days of program
participation caused students to score higher on achievement tests. A correlational finding
between more days of program attendance and higher student achievement may instead explain
the characteristics of participating students. A correlation may exist because students who enjoy
school may be more likely to achieve higher assessment scores, and students who enjoy school
may be more likely to participate in programming that is similar to their school day activities—
that is, they may have higher levels of attendance in the 21st CCLC programs.
Taken together, the findings for within-program analyses are useful in exploring particular
student or center characteristics associated with lower (or higher) levels of student academic and
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—88
behavioral outcomes. The reader should keep in mind that these findings are purely descriptive
in nature and do not in any way imply that a given center or student characteristic is causally
related to a given outcome.
Impact of 21st CCLC Participation on Student Achievement
Although the within-program analyses were correlational in nature, steps were taken to also run a
series of causal models to assess the impact of the 21st CCLC on a variety of youth outcomes in
2011–12. In order to construct such causal estimates, the evaluation team employed a quasi-
experimental research design to examine the effect of participating in 21st CCLC programming
on student reading and mathematics achievement; cumulative GPA; percent of credits earned;
and one nonacademic measure: number of unexcused absences. Students’ reading and math
achievement were measured by the Washington state exam for Grades 3–8, the Measurements of
Student Progress (MSP) and the state exam for high school students, High School Proficiency
Exam (HSPE). The goal of this analysis was to answer the following evaluation questions:
To what extent is there evidence that students participating in services and activities
funded by 21st CCLC demonstrated better performance on reading and mathematics
assessments, cumulative GPA, and percentage of credits earned as compared with similar
students not participating in the program?
To what extent is there evidence that there are differences between students participating
in services and activities funded by 21st CCLC and similar students not participating in
the program in terms of the number of unexcused absences?
Specifically, using a propensity score stratification approach, the study compared the
performance of students who participated in 21st CCLC with similar students who did not
participate. Participation was defined two ways for the purpose of the analysis. First, students
who attended at least 30 days were compared with students who attended 0 days. Second,
students who attended at least 60 days were compared with students who attended 0 days. These
definitions of treatment were determined to ensure that the comparison of program effect was
based on students who received a significant dose of 21st CCLC programming.
In any evaluation of a program where participants are not randomly assigned to participate in the
program, the problem of selection is paramount. We know that it is likely that students who
participate in 21st CCLC programming are different from those who do not attend. These
differences can bias estimates of program effectiveness because they make it difficult to
disentangle preexisting differences between students who attended the program and those who
did not, from the effect of attending the program. In general, we found that students who
attended the program tended to be lower achieving students than those who did not, prior to the
start of the current academic year. The quasi-experimental approach outlined here, propensity
score matching (PSM), is a method for mitigating the existing bias in program effect (i.e., if one
were to simply compare the students who attended and those who did not).
PSM is a two-stage process designed to address this problem. In the first stage, the probability
that each student participates in the 21st CCLC program was modeled on available observable
characteristics. By modeling selection into the program, this approach allowed us to compare
participating and nonparticipating students who would have had a similar propensity to select
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—89
into the program based on observable characteristics that were available in the data received
from the state of Washington. In the second stage, the predicted probability of participation was
used to model student outcomes while accounting for selection bias. We balanced pretreatment
group differences in observed covariates using a propensity score stratification and marginal
mean weighting approach (Hong & Hong, 2009).
Stage 1: Creation of the Comparison Group
The outcome of interest in modeling propensity scores is treatment status (1 for students
participating in the program, 0 for the comparison group). To account for this binary outcome,
logistic regression was used to model the logit (or log-odds) of student group assignment status.
Examples of student-level variables used to fit the propensity score models included:
Prior achievement in reading and math
Student demographic information including
Gender
Racial status
Language of origin
Socioeconomic status
Special education status
Migrant status
Immigrant status
School type
In addition to the student-level variables, the propensity score model also included school
variables that added information about the school a student attended (to account for school-based
contextual differences, which may account for differences in the propensity for a student to
participate). A total of 123 variables were considered for the propensity score model. Data were
not available for each of these covariates for all students. To account for this, indicator variables
were used to model the relationship between the pattern of missing data and propensity to
participate in the program (Rosenbaum & Rubin, 1984). The propensity score model was fit
separately for each grade (Grades 3–12), and separately for each definition of treatment (30+
days; 60+ days). The final propensity score models for each grade were checked to ensure that
the analysis sample was balanced across relevant covariates. The propensity score models all
produced comparison samples that were balanced with the treatment across the 123 variables
examined for balance. This result indicates that the treatment and comparison groups had no
significant differences from one another (prior to treatment) as measured by these 123 variables.
Impact Analysis Results
Tables 33a and 33b show the effect of 21st CCLC programming on student reading and
mathematics achievement, cumulative GPA, percentage of credits earned, and number of
unexcused absences, pooled across grade levels (for both 30+ day and 60+ day treatment
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—90
definitions). It is important to note that the comparison group for the 30+ day and 60+ day
treatment definitions will differ. Separate propensity score models were fit for each, and it is
reasonable to think that students who attend 60 or more days are different from those who only
attend 30 or more days.
As shown in Table 33a, a statistically significant, positive impact of 21st CCLC was found for
reading achievement at the 0.01 significance level for both 30+ day and 60+ day treatments, with
students in treatment group achieving 0.027 standardized deviation units higher for the 30+ day
treatment and 0.033 standardized deviation units higher for the 60+ day treatment than students
in the comparison group. There was also a significant positive impact of 21st CCLC
programming on student mathematics achievement at the 0.01 significance level for both 30+
day and 60+ day treatments, with students in the treatment group achieving 0.044 standardized
deviation units higher for the 30+ day treatment and 0.035 standardized deviation units higher
for the 60+ day treatment than students in the comparison group. Although positive, these
program effects are quite small.
In terms of outcomes pertaining to high school students, there was a nonsignificant negative
impact of 21st CCLC on student cumulative GPA for the 30+ day treatment and significant
positive impact for the 60+ day treatment. For the 60+ day treatment, the cumulative GPA in the
treatment group was 0.195 standardized deviation units higher than that in the comparison group.
For percentage of credits earned, a significant positive impact was found at 60+ days of
participation. Regardless the significance of effect estimates all effect sizes are small (Cohen,
1988).
Table 33a. Impact of 21st CCLC on Achievement Pooled Across Grades
Subject Treatment Effect Size S.E.1 of Effect Size p
Reading2
30+ day 0.027 0.008 0.001
60+ day 0.033 0.011 0.004
Math3 30+ day 0.044 0.008 <0.001
60+ day 0.035 0.011 0.002
Cumulative GPA4
30+ day -0.022 0.026 0.399
60+ day 0.195 0.049 <0.001
Percent of credits earned4
30+ day 0.034 0.027 0.212
60+ day 0.144 0.048 0.003
1 Standard error
2 Include Grades 4–8, 10
3 Include Grades 4–8
4 Include Grades 9–12
In terms of unexcused absences, a statistically significant, negative effect of 21st CCLC was
found for the number of unexcused absences at the 0.01 significance level for both 30+ day and
60+ day treatments. In this regard, the number of unexcused absences was lower in the treatment
group than that in the comparison group at both 30 and 60 days of 21st CCLC. These effects
were moderate to large. More specifically, for the 30+ day treatment groups, the number of
unexcused absences in the treatment group was 66 percent of the level found in the comparison
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—91
group made up of nonparticipating students. For the 60+ day treatment group, the number of
unexcused absences in the treatment group was just 39 percent of the level found in the
comparison group.
Table 33b. Impact of 21st CCLC on Number of Unexcused Absences
Pooled Across Grades
Treatment Effect S.E. p Weighted Mean Ratio (Treatment/Comparison)1
30+ days -0.312 0.009 <0.001 0.657
60+ days
-0.638 0.017 <0.001 0.393
1Weighted Mean Ratio is the ratio of mean of unexcused absences after accounting the weight
generated for each student during PSM process.
Tables 34a and 34b show the impact on achievement and number of unexcused absences broken
down by grade for the 30+ day treatment group. In terms of reading achievement, there was a
statistically significant, positive impact of treatment on reading achievement for Grades 6 and 7,
and no significant impact for all other grades. There also was a significant positive impact of
treatment on mathematics achievement for Grades 6 and 8.
In terms of high school outcomes, a significant positive impact of 21st CCLC was found on
student cumulative GPA for Grade 11, and a significant negative impact was found for Grade 9.
Of some interest in Table 34a was that 21st CCLC participation did not have significant impact
on the percentage of credits student earned at a single grade, although as shown in Table 33a, a
significant impact was seen across all high school grades levels at 60 days of participation.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—92
Table 34a. Impact of 21st CCLC on Achievement – 30+ Day Treatment
Grade Reading Math Cumulative GPA Percent of Credits Earned
Effect S.E. p Effect Size Effect S.E. p Effect Size Effect S.E. p Effect Size Effect S.E. p Effect Size
4 1.813 5.426 0.745 0.043 -0.240 10.727 0.983 -0.004
5 1.138 0.801 0.156 0.029 0.236 0.957 0.805 0.005
6 1.694 0.598 0.005 0.038 3.671 0.739 <0.001 0.066
7 1.459 0.768 0.057 0.031 1.338 0.933 0.152 0.022
8 0.842 0.880 0.339 0.016 3.833 0.909 <0.001 0.065
9 -0.189 0.047 <0.001 -0.186 -0.005 0.011 0.646 -0.024
10 -0.841 1.752 0.631 -0.016 0.006 0.043 0.895 0.006 0.013 0.010 0.189 0.060
11 0.099 0.045 0.028 0.121 0.014 0.011 0.196 0.071
12 0.039 0.053 0.470 0.052 0.003 0.012 0.834 0.016
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—93
In terms of unexcused absences, a statistically significant, negative impact was found for the
number of unexcused absences for all grades (3–12) at 30 days of participation (again, a negative
result is desired here because it indicates fewer unexcused absences). Table 34b indicates that the
number of unexcused absences in the treatment group was lower for all grades than in the
comparison group, with the largest impact happening in Grades 5–8.
Table 34b. Impact of 21st CCLC on Number of Unexcused Absences,
30+ Day Treatment
Grade Number of Unexcused Absences
Effect S.E. p Weighted Mean Ratio (Treatment/Comparison)
3 -0.376 0.037 <0.001 0.736
4 -0.387 0.041 <0.001 0.715
5 -0.550 0.042 <0.001 0.566
6 -0.459 0.028 <0.001 0.621
7 -0.394 0.027 <0.001 0.631
8 -0.444 0.028 <0.001 0.665
9 -0.185 0.032 <0.001 0.784
10 -0.350 0.021 <0.001 0.702
11 -0.092 0.024 <0.001 0.916
12 -0.095 0.029 0.001 0.853
Tables 35a and 35b are similar to Tables 34a and 34b, but they show the results for those
participating in 21st CCLC for 60+ days. A significant positive impact of 21st CCLC on reading
achievement was found for youth in Grade 6 (see Table 35a). In addition, there was a significant
positive impact of 21st CCLC on math achievement for youth in Grades 6 and 8 at the 60-day
threshold.
In terms of outcomes related to high school students, a significant positive impact was found on
student cumulative GPA for youth in Grades 10 and 11, and a significant negative impact was
found for youth in Grade 9. Also as shown in Table 35a, there was a significant positive impact
of 21st CCLC on the percentage of credits students earned for Grade 10 in particular. There are
some interesting findings here, particularly between Grades 9 and 10, which may warrant some
future investigation given the increasing effort being dedicated to ensuring students make an
effective transition to high school in Grade 9.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—94
Table 35a. Impact of 21st CCLC on Achievement – 60+ Day Treatment
Grade Reading Math Cumulative GPA Percent of Credits Earned
Effect S.E. p Effect Size Effect S.E. p Effect Size Effect S.E. p Effect Size Effect S.E. p Effect Size
4 5.552 7.037 0.474 0.129 -3.785 9.492 0.710 -0.059
5 1.305 0.971 0.179 0.032 0.280 1.184 0.813 0.005
6 1.870 0.860 0.030 0.040 3.807 1.083 0.000 0.066
7 -0.103 1.305 0.937 -0.002 0.394 1.548 0.799 0.006
8 2.279 1.592 0.152 0.042 3.439 1.682 0.041 0.056
9 -0.192 0.093 0.040 -0.192 0.000 0.021 0.986 -0.002
10 4.074 2.865 0.155 0.077 0.278 0.073 0.000 0.278 0.066 0.016 <0.001 0.297
11 0.512 0.099 <0.001 0.652 0.022 0.020 0.281 0.117
12 0.135 0.102 0.186 0.189 -0.005 0.021 0.801 -0.031
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—95
As shown in Table 35b, a statistically significant, negative impact was found for the number of
unexcused absences for all grades (3–12) when treatment was defined as 60 days of participation
in 21st CCLC. Here again, the number of unexcused absences in treatment group was lower for
all grades than in the comparison group, with the largest impact happening in Grades 5–8.
Table 35b. Impact of 21st CCLC on Number of Unexcused Absences,
60+ Day Treatment
Grade Number of Unexcused Absences
Effect S.E. p Weighted Mean Ratio (Treatment/Comparison)
3 -0.482 0.045 <0.001 0.628
4 -0.478 0.055 <0.001 0.657
5 -0.805 0.060 <0.001 0.439
6 -0.947 0.054 <0.001 0.369
7 -0.950 0.056 <0.001 0.377
8 -0.653 0.053 <0.001 0.510
9 -0.161 0.062 0.010 0.794
10 -0.987 0.051 <0.001 0.420
11 -0.499 0.054 <0.001 0.503
12 -0.241 0.066 <0.001 0.691
Summary of Impact Analyses Results
Generally, findings from the impact analyses conducted in relation to youth outcomes associated
with the 2011–12 project period indicated positive program impacts across each of the outcomes
examined:
Significant, positive program impacts were found for both reading and mathematics at
both the 30-day and 60-day participation thresholds. Findings from the Year 1 report
included such effects for mathematics only and not for reading. However, effect sizes
were very small, ranging from .027 for reading at 30 days to .044 for mathematics at 30
days.
Significant, positive program impacts were found for both cumulative GPA and credits
earned / credits attempted at only the 60-day participation threshold. The effect size for
cumulative GPA at 60 days was .195, a small effect, and the effect size for credits earned
/ credits attempted at 60 days was .144, also a small effect.
Significant, positive program impacts were found in terms of a lower number of
unexcused absences at both the 30-day and 60-day participation threshold. The effect size
at 30 days was -.312, a moderate effect, and the effect size at 60 days was -.638, a large
effect.
It is important to note that the propensity score stratification approach employed here seeks to
minimize the impact of selection bias on the estimates of program impact. However, it is an
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—96
untestable assumption that such models can fully account for selection bias. To the extent that
other variables exist (not available for this analysis) that predict student participation in 21st
CCLC and are also related to student achievement or unexcused absences, these analyses may be
limited. To that end, these analyses provide initial evidence about the impact of 21st CCLC on
academic achievement and unexcused absences, but should not necessarily be considered
equivalent to experimental studies which have strong internal validity.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—97
Conclusions
In conducting the 2011–12 statewide evaluation of the Washington 21st CCLC-funded
programming, a primary goal was to understand how and to what degree centers were
implementing research-supported practices in their programming, as well as what impact
participation in 21st CCLC-funded programming had on student outcomes such as reading and
math achievement, grade point average, and absences. Specifically, the evaluation questions that
guided the design are as follows:
1. What were the primary characteristics associated with the grants and centers funded by
21st CCLC and the student population served by the program?
2. To what extent was there evidence that centers funded by 21st CCLC implement
research-supported practices related to quality afterschool programming?
3. To what extent is there evidence of a relationship between center and student
characteristics and the likelihood that students demonstrated better performance on
program attendance and youth outcomes, with a particular emphasis on exploring the
relationship between leading indicator status and these outcomes?
4. To what extent is there evidence that students participating in services and activities
funded by 21st CCLC demonstrated better performance on youth outcomes as compared
with similar students not participating in the program?
The 2011–12 evaluation built on the previous year by further developing the leading indicators
initiated in Year 1. The leading indicator system is designed provide grantees with data from the
evaluation to allow them to assess how they have adopted research-supported best practices,
what their strengths and weaknesses are, and how they might improve programming moving
forward. The leading indicators provide the structure for the correlational within-program
analysis, allowing for clustering of similar performance groups and examination of the
association between performance level on the indicators and youth outcomes.
In considering the relationship between the leading indicators and youth outcomes as examined
in Evaluation Question 3, it was hypothesized that centers scoring below average on key
indicators would be negatively associated with youth outcomes. This hypothesis was supported
for centers in the Instructional Context-Content Below Average cluster, which includes indicators
such as intentionality in program design, linkages to the school day, and use of student data to
inform programming. In this case, there was a negative association between centers with low
scores on these indicators and student performance on state reading assessments and a positive
association with unexcused absences. Similarly, membership in the Organizational Context
Below Average cluster, which includes training, program climate, and reported internal
communication, was negatively associated with youth attendance in the 21st CCLC program.
However, this analysis also revealed findings that did not support this hypothesis. Membership in
the Organizational Context Below Average cluster was positively associated with cumulative
GPA and credits earned, and membership in the Instructional Context-Process Below Average
cluster was positively associated with higher levels of program attendance, although this may be
due to a relatively large number of centers serving elementary students only in this cluster, as
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—98
elementary school students tend to have higher levels of program attendance than middle or high
school students.
In addition to membership in below-average clusters for the leading indicators, various student-
and center-level characteristics were examined with relation to student outcomes. Analysis
revealed that a positive relationship existed between the number of days of participation in 21st
CCLC programming and academic performance outcomes, and a negative association existed
with unexcused absences from school. In addition, centers with school-based status were
negatively associated with youth performance on math and reading assessments and positively
associated with unexcused absences. Several student characteristics were also found to have a
negative association with academic performance, including eligibility for free or reduced-price
lunch, receipt of special education services, limited English proficiency, and minority status.
With regards to the final evaluation question, the evaluation also examined how program
participation impacted youth outcomes using a propensity score matching approach to reduce
selection bias. Exploring the impact of participation in the 21st CCLC programming by
comparing participants with nonparticipants revealed positive program impacts when pooled
across all grades, although not necessarily for youth at all grade levels. The analysis revealed
small but significant impacts on both reading and math achievement when pooled across grades,
whereas the findings for Year 1 revealed positive impacts for math only.
There were also significant but small positive effects of the 21st CCLC program on cumulative
GPA and percentage of credits earned for students in the 60-day treatment group, and students in
both the 30-day and 60-day treatment groups showed significantly lower levels of unexcused
absences than nonparticipants. In the case of program effects on unexcused absences, program
effects were moderate to large. This analysis provides a basis for continued exploration of the
impact of 21st CCLC programming on participants, especially in divergent impacts on individual
grade levels.
This report’s findings on leading indicators, correlational relationships, and impact analyses
provide guidance for grantees on areas for continued growth in the upcoming years, including (1)
using data to inform services for individual students, (2) allowing staff more time for planning
and preparation, and (3) identifying ways to incorporate more youth ownership into the program
at grade-appropriate levels. These results are very similar to those identified in the Year 1 report.
In addition, there appears to be some evidence that (a) there are opportunities for growth in terms
of how centers go about designing and delivering activities from a content perspective and (b)
that enhanced levels of practice in this area are related to better school-related outcomes.
Although OSPI has an infrastructure for supporting instructional quality from a process
perspective, it may want to give consideration to the types of supports it could provide to
enhance the manner in which 21st CCLC supports the cultivation of skills and knowledge from a
content perspective, particularly in relation to the needs of participating youth.
Although a variety of positive program effects were demonstrated in this year, OSPI is interested
in further exploring the types of impacts 21st CCLC is having on social-emotional learning, 21st
century skills and competencies, and noncognitive outcomes. Toward this end, in Year 3 of the
evaluation, AIR will be working to collect information from grantee project directors on what
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—99
they believe their programs are impacting in these areas and what their priorities should be in
terms of testing measurement strategies to assess program impact on such outcomes. Steps will
be taken to select a sample of instruments designed to measure high-priority outcomes and pilot
those in a small number of centers during the spring semester of the 2013–14 school year.
Finally, the leading indicators represent a substantial investment of time and effort to provide
Washington 21st CCLC grantees with actionable data to guide and support program
improvement efforts. A key goal of the Year 3 evaluation will be to better understand the
efficacy of these tools as a vehicle for supporting quality improvement efforts and to highlight
portions of the system that are proven to have especially high value to grantees and OSPI.
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—100
References
Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). Shared features of high-
performing after-school programs: A follow-up to the TASC evaluation. Washington DC:
Policy Studies Associates.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (second edition).
Hillsdale, NJ: Lawrence Erlbaum Associates.
Durlak, J. A., & Weissberg R. P. (2007) The impact of after-school programs that promote
personal and social skills. Chicago: Collaborative for Academic, Social, and Emotional
Learning.
Eccles, J., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth
development. Washington, DC: National Academies Press.
Glisson, C. (2007). Assessing and changing organizational culture and climate for effective
services. Research on Social Work Practice, 17.
Granger, R. C., Durlak, J. A., Yohalem, N., & Reisner, E. (2007). Improving after-school
program quality. New York: William T. Grant Foundation.
Granger, R. (2008). After-school programs and academics: Implications for policy, practice, and
research. Social Policy Report, 22(2), 3–19. Ann Arbor, MI: Society for Research in
Child Development. Retrieved from
http://www.srcd.org/documents/publications/spr/spr22-2.pdf
Hong, J., & Hong, Y. (2009). Reading instruction time and heterogeneous grouping in
kindergarten: An application of marginal mean weighting through stratification.
Educational Evaluation and Policy Analysis, 31(1), 54–81.
Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. A., Snow, D., & Martin-Glenn, M.
(2006). Out-of-school-time programs: A meta-analysis of effects for at-risk students.
Review of Educational Research, 76(2), 275–313.
Little, P. (2007). The quality of school-age child care in after-school settings (Research to Policy
Connections, No. 7). New York: National Center for Children in Poverty, Columbia
University Mailman School of Public Health.
Naftzger, N., Bonney, C., Donahue, T., Hutchinson, C., Margolin, J., & Vinson, M. (2007). 21st
Century Community Learning Centers (21st CCLC) analytic support for evaluation and
program monitoring: An overview of the 21st CCLC performance data: 2005–06.
Naperville, IL: Learning Point Associates. Retrieved from
http://www2.ed.gov/programs/21stcclc/2501yr3ppics.doc
Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using
subclassification on the propensity score. Journal of the American Statistical Association,
79(387).
American Institutes for Research Washington 21st CCLC Year 2 Evaluation—101
Smith, C. (2007, March). Predictors of quality at the point of service provision: Empirical and
methodological background for the YPQA field trial. Presented at the biennial meeting of
the Society for Research in Child Development, Boston.
Vandell, D. L., Reisner, E. R., Brown, B. B., Dadisman, K., Pierce, K. M., Lee, D., et al. (2005).
The study of promising after-school programs: Examination of intermediate outcomes in
Year 2. Madison, WI: Wisconsin Center for Education Research. Retrieved from
http://childcare.wceruw.org/pdf/pp/year2_executive_summary_and_brief_report.pdf
Wilson-Ahlstrom, A., & Yohalem, N. (with Pittman, K.). (2007). Building quality improvement
systems: Lessons from three emerging efforts in the youth-serving sector. Washington,
DC: Forum for Youth Investment, Impact Strategies.
Yohalem, N., Wilson-Ahlstrom, A., Fischer, S., & Shinn, M. (2009). Measuring youth program
quality: A guide to assessment tools (2nd ed.). Washington, DC: Forum for Youth
Investment.
Zief, S. G., Lauver, S., & Maynard, R. A. (2006). The impacts of after-school programs on
student outcomes. Campbell Systematic Review, 3, 1–52. Retrieved from
http://www.campbellcollaboration.org/lib/download/58/
American Institutes for Research Washington 21st CCLC Program Evaluation: Year 1
LOCATIONS
Domestic
Washington, D.C.
Atlanta, GA
Baltimore, MD
Chapel Hill, NC
Chicago, IL
Columbus, OH
Frederick, MD
Honolulu, HI
Indianapolis, IN
Naperville, IL
New York, NY
Portland, OR
Sacramento, CA
San Mateo, CA
Silver Spring, MD
Waltham, MA
International
Egypt
Honduras
Ivory Coast
Kenya
Liberia
Malawi
Pakistan
South Africa
Zambia
ABOUT AMERICAN INSTITUTES FOR RESEARCH
Established in 1946, with headquarters in Washington, D.C.,
American Institutes for Research (AIR) is an independent,
nonpartisan, not-for-profit organization that conducts behavioral
and social science research and delivers technical assistance
both domestically and internationally. As one of the largest
behavioral and social science research organizations in the world,
AIR is committed to empowering communities and institutions with
innovative solutions to the most critical challenges in education,
health, workforce, and international development.
14-1167_01/14
20 North Wacker Drive, Suite 1231
Chicago, IL 60606-2901
312.288.7600 | Fax: 312.288.7601
www.air.org