AmeriCorps Technical Review Cover Sheet...Dec 21, 2020 · Section B Program Design (0-30 points...
Transcript of AmeriCorps Technical Review Cover Sheet...Dec 21, 2020 · Section B Program Design (0-30 points...
-
1
2021 New York State AmeriCorps Competitive Application Review Process
REVIEWER WORKSHEET
(The individual reviewer worksheet does not determine final consensus scoring)
Proposal Name:
Reviewer Name (Print):
Section Totals
Section A Executive Summary: (Required) SUBTOTAL: YES / NO
Section B Program Design: Theory of Change and Logic Model (0-24) ______ + EVIDENCE BASE + Notice Priority (0) + Member Experience (0-6) ______
Note: Evidence Base (20 points) will be reviewed separately by OCFS Strategic Planning and Policy Development staff on 2021 Evidence Review Tool = SUBTOTAL: ________ (0-30)
Section C Organizational Capability: Organization Background and Staffing (0-9) ______ + Compliance and Accountability (0-8) ______ + Culture that Values Learning (0-4) ______ +
Member Supervision (0-4) ______ = SUBTOTAL: ________(0-25)
Section D Cost Effectiveness & Budget Adequacy: Cost Effectiveness and Budget Adequacy (0-25)
= SUBTOTAL: ________ (0-25)
Section E Evaluation Plan: (Required for recompeting grantees only) YES / NO
APPLICATION SCORE: ________(0-80)
-
2
Section A: Executive Summary Executive Summary (No points) Comments:
• Did the applicant provide an ExecutiveSummary? Yes No
Section B: Program Design (8 subsections)1.Theory of Change and Logic Model (0-24 points) Comments:
The Theory of Change Shall address:
• Did the applicant clearly describe that theproposed intervention is responsive to theidentified community problem? Yes No
• Did the applicant clearly describe theproposed intervention is clearly articulatedincluding the design, dosage, targetpopulation, and roles of AmeriCorpsmembers and (if applicable) leveragedvolunteers?
Yes No
• Did the applicant clearly describe how theintervention is likely to lead to the outcomesidentified in the applicant’s theory ofchange?
Yes No
-
3
Section B: Program Design (CONTINUED)
• Did the applicant clearly describe how theexpected outcomes articulated in theapplication narrative and logic modelrepresent meaningful progress inaddressing the community problemidentified by the applicant?
Yes No
• Did the applicant clearly describe therationale for utilizing AmeriCorps membersto deliver the intervention(s) is reasonable?
Yes No
• Did the applicant clearly describe theservice role of AmeriCorps members willproduce significant contributions to existingefforts to address the stated problem?
Yes No
Note: The logic model is a visual representation of the applicant’s theory of change. Programs may include short, medium or long-term outcomes in the logic model. Applicants are not required to measure all components of their theory of change. The applicant’s performance measures should be consistent with the program’s theory of change and should represent significant program activities.
In the application narrative, applicants should discuss their rationale for setting output and outcome targets for their performance measures.
Rationales and justifications should be informed by the organization’s performance data (e.g., program data observed over time that suggests targets are reasonable), relevant research (e.g., targets documented by organizations running similar programs with similar populations), or prior program evaluation findings.
Applicants with multiple interventions should complete one Logic Model chart which incorporates each intervention. Logic model content that exceeds three pages will not be reviewed.
-
4
Section B: Program Design (CONTINUED) The Logic Model shall depict:
• Did the applicant provide a summary of thecommunity problem. Yes No
• The inputs or resources that are necessaryto deliver the intervention, including but notlimited to:o Locations or sites in which members
will provide serviceso Number of AmeriCorps members that
will deliver the intervention
Yes No
• The core activities that define theintervention or program model thatmembers will implement or deliver,including:o The duration of the intervention (e.g.,
the total number of weeks, sessions ormonths of the intervention).
o The dosage of the intervention (e.g.,the number of hours per session orsessions per week.)
o The target population for theintervention (e.g., disconnected youth,third graders at a certain readingproficiency level).
Yes No
• The measurable outputs that result fromdelivering the intervention (i.e. number ofbeneficiaries served types and number ofactivities conducted.) If applicable, identifywhich National Performance Measures willbe used as output indicators.
Yes No
-
5
• Outcomes that demonstrate changes inknowledge/skill, attitude, behavior, orcondition that occur as a result of theintervention. If applicable, identify whichNational Performance Measures will beused as outcome indicators.
Yes No
Theory of Change and Logic Model (0-24 points) Total Points Awarded: _________
Section B: Program Design (CONTINUED) 2. Evidence BaseNote: Evidence Base (20 points) will be reviewed separately by OCFS Strategic Planning and Policy Development staff on 2021 Evidence Review Tool
3. Notice Priority (0 points) Comments: • Did the applicant clearly describe how its
proposed program is within one or more of the2021 AmeriCorps Funding Priorities section ofthe 2021 Notice of Funding Availability andmore fully described in the 2021 MandatorySupplemental Guidance and the proposedprogram meets all of the requirements detailedin the Funding Priorities section and in the2021 Mandatory Supplemental Guidance.
Yes No
Notice Priority (0 points allowable)
4. Member Experience (0-6 points) Comments:
• Did the applicant clearly describe how theAmeriCorps members will gain skills as aresult of their training and service that canbe utilized and will be valued by futureemployers after their service term iscompleted?
No Yes
-
6
• Did the applicant clearly describe how theprogram will recruit AmeriCorps membersfrom the geographic or demographiccommunities in which the programoperates?
Yes No
• Did the applicant clearly describe how theapplicant will foster and inclusive serviceculture where different backgrounds,talents, and capabilities are welcomed andleveraged for learning and effective servicedelivery?
Yes No
Member Experience (0-6 points) Total Points Awarded: _________
Section B Program Design (0-30 points allowable for all 5 subsections) TOTAL POINTS AWARD: ________
Section C: Organizational Capacity (4 subsections) 1. Organizational Background and Staffing (0-9) Comments:
• Did the applicant clearly describe the roles,responsibilities, and structure of the staff thatwill be implementing the AmeriCorps programas well as providing oversight and monitoringfor the program?
Yes No
Organizational Background and Staffing (0-9 allowable) Total Points Awarded: _________ 2. Compliance and Accountability (0-8) Comments: • Did the applicant clearly describe a monitoring
and oversight plan to prevent and detect non-compliance and enforce compliance withAmeriCorps rules and regulations includingthose related to prohibited and unallowableactivities and criminal history checks atgrantee, subgrantee (if applicable), andservice site locations?
Yes No
-
7
• Did the applicant provided CNCS-requiredevaluation report that meets CNCSrequirements (if applicable)?
Yes No
• Did the applicant provided CNCS-requiredevaluation report is of satisfactory quality (ifapplicable)?
Yes No
Note: Commission of National and Community Service will provide the reviewers with the CNCS-required evaluation report requirement for each application because the it various based on how many years has program been operational. OCFS subject matter expert will also provide feedback and comment whether if the evaluation reports meet CNCS requirement and/or they are of satisfactory quality. The reviewers the final determination on score based on the information provided above.
Compliance and Accountability (0-8 allowable) Total Points Awarded: _________
Section C: Organizational Capacity (CONTINUED) 3. Culture that Values Learning (0-4) Comments:
• Did the applicant clearly describe how theapplicant’s board, management, and staffcollects and uses information for learningand decision making?
Yes No
Culture that Values Learning (0-4 allowable) Total Points Awarded: _________ 4. Member Supervision (0-4) Comments:
• Did the applicant clearly describe how theAmeriCorps members will receive sufficientguidance and support from their supervisorto provide effective service?
Yes No
-
8
• Did the applicant clearly describe how theAmeriCorps supervisors will be adequatelytrained/prepared to follow AmeriCorps andprogram regulations, priorities, andexpectations?
Yes No
Member Supervision (0-4 allowable) Total Points Awarded: _________
Section C Organizational Capability (0-25 points allowable for 4 subsections) TOTAL POINTS AWARD: ________
Section D: Cost Effectiveness and Budget Adequacy (1 subsection)Note: Reviewers will assess the quality of the application’s budget to the following criteria below. Do not assume all sub-criteria are of equal value. This criterion will be assessed based on the budget submitted. No narrative should be entered in the narrative box except for “See budget.”
1. Cost Effectiveness and Budget Adequacy (0-25 points) Comments:
• Is the budget submitted withoutmathematical errors and proposed costs areallowable, reasonable, and allocable to theaward?
Yes No
• Is the budget submitted with adequateinformation to assess how each line item iscalculated?
Yes No
• Is the budget submitted in compliance withthe budget instructions? Yes No
• Is the match information submitted withadequate information to support the amountwritten in the budget?
Yes No
-
9
• Is the budgeted match equal to or morethan the required match for the givenprogram year?
• Is the cost per MSY equal or less than themaximum cost per MSY stated in the 2021-22 NYS AmeriCorps Competitive RFP?
Yes No
• Did the applicant clearly provide the currentindirect rate cost if used to claim indirect /administrative costs?
Yes No
Section D: Cost Effectiveness and Budget Adequacy (CONTINUED)
• Did the applicant clearly Identify the non-CNCS funding and resources necessary tosupport the project, including for FixedAmount applicants?
Yes No
• Did the applicant clearly Indicate theamount of non-CNCS resourcecommitments, type of commitments (in-kindand/or cash), the sources of thesecommitments, and if the commitments areproposed or secured?
Yes No
Cost Effectiveness (0-25) Total Points Awarded: _________
Section D Cost Effectiveness and Budget Adequacy (0-25 points allowable for 1 subsection) TOTAL POINTS AWARD: ________
Yes No
-
DRAFT
Page 1
STATE APPLICATION IDENTIFIER:
FEDERAL IDENTIFIER:
21ES231071
2b. APPLICATION ID:
24-NOV-20
3. DATE RECEIVED BY STATE:
4. DATE RECEIVED BY FEDERAL AGENCY:
City Year, Inc.LEGAL NAME:
287 Columbus Avenue
Erin E. BennettNAME:
(617) 927-2373
TELEPHONE NUMBER:
(617) 927-2520 FAX NUMBER:
[email protected] E-MAIL ADDRESS:
2228825496. EMPLOYER IDENTIFICATION NUMBER (EIN): 7. TYPE OF APPLICANT:
8. TYPE OF APPLICATION (Check appropriate box).
If Amendment, enter appropriate letter(s) in box(es):
94.00610a. CATALOG OF FEDERAL DOMESTIC ASSISTANCE NUMBER:
AmeriCorps Fixed Amount Grant (State)10b. TITLE: City Year New York FPG
11.a. DESCRIPTIVE TITLE OF APPLICANT'S PROJECT:
New York City, NY - Schools based in Bronx, Brooklyn, Manhattan, and Queens
12. AREAS AFFECTED BY PROJECT (List Cities, Counties, States, etc):
07/01/21START DATE: 06/30/22END DATE: MA 07
$ 4,156,500.00a. FEDERAL
$ 0.00b. APPLICANT
$ 0.00c. STATE
$ 0.00d. LOCAL
$ 0.00e. OTHER
$ 4,156,500.00g. TOTAL
Jessica Greenfield
a. TYPED NAME OF AUTHORIZED REPRESENTATIVE: b. TITLE:
(617) 927-2318
c. TELEPHONE NUMBER:
11/16/20
e. DATE SIGNED:
Service/Civic OrganizationCommunity-Based Organization National Non-Profit (Multi-State)
2a. DATE SUBMITTED TO CORPORATION FOR NATIONAL AND COMMUNITY SERVICE (CNCS):
1. TYPE OF SUBMISSION:
Non-Construction
5. APPLICATION INFORMATION
NAME AND CONTACT INFORMATION FOR PROJECT DIRECTOR OR OTHER PERSON TO BE CONTACTED ON MATTERS INVOLVING THIS APPLICATION (give area codes):
ADDRESS (give street address, city, state, zip code and county):
A. AUGMENTATION B. BUDGET REVISION
C. NO COST EXTENSION D. OTHER (specify below):
9. NAME OF FEDERAL AGENCY:
Corporation for National and Community Service
13. PROPOSED PROJECT: 14. CONGRESSIONAL DISTRICT OF:
15. ESTIMATED FUNDING: 16. IS APPLICATION SUBJECT TO REVIEW BY STATE EXECUTIVE ORDER 12372 PROCESS?
YES. THIS PREAPPLICATION/APPLICATION WAS MADE AVAILABLE TO THE STATE EXECUTIVE ORDER 12372 PROCESS FOR REVIEW ON:
DATE:
17. IS THE APPLICANT DELINQUENT ON ANY FEDERAL DEBT?YES if "Yes," attach an explanation. NOX
18. TO THE BEST OF MY KNOWLEDGE AND BELIEF, ALL DATA IN THIS APPLICATION/PREAPPLICATION ARE TRUE AND CORRECT, THE DOCUMENT HAS BEEN DULY AUTHORIZED BY THE GOVERNING BODY OF THE APPLICANT AND THE APPLICANT WILL COMPLY WITH THE ATTACHED ASSURANCES IF THE ASSISTANCEIS AWARDED.
APPLICATION FOR FEDERAL ASSISTANCE
Boston MA 02116 - 5114
NEW
CONTINUATION AMENDMENT
X
7b.
7a. Non-Profit
f. PROGRAM INCOME $ 0.00
PART I - FACE SHEET
DUNS NUMBER: 622374122
Modified Standard Form 424 (Rev.02/07 to confirm to the Corporation's eGrants System) Application
NEW/PREVIOUS GRANTEE
11.b. CNCS PROGRAM INITIATIVE (IF ANY):
X NO. PROGRAM IS NOT COVERED BY E.O. 12372
d. SIGNATURE OF AUTHORIZED REPRESENTATIVE:
Year #:
X
1
a.Applicant b.Program NY 10
County: Suffolk
UEI NUMBER:
-
DRAFT
Page 2
For Official Use Only
Narratives
Executive Summary
Rationale and Approach/Program Design
City Year New York proposes to have 255 full-time AmeriCorps members who will provide full-day
education services to students in 20 public schools in New York City (NYC). At the end of the first
program year, as Student Success Coaches, the AmeriCorps members will mentor 2,550 students, 322
of whom will improve literacy skills and 312 will improve math skills; and, all will be supported in the
development of social-emotional skills that improve school engagement. Members will also provide
extended day and school climate activities. In addition, the AmeriCorps members will leverage 200
volunteers who will be engaged in service projects in our school communities. This program will focus
on the CNCS focus area of Education. The CNCS fixed-price investment of $4,156,500 will leverage
$4,156,500 in city funding.
1.THEORY OF CHANGE & LOGIC MODEL: RESPONSIVE TO COMMUNITY NEED: Students
who are at risk to drop out of school before graduation can be identified as early as third grade by
performance in the commonly accepted early warning indicators of academic performance in English
and math, attendance, and behavior. Within City Year New York's (CYNY) current school partners,
prior to COVID-19 closures and distance learning, 61% of 3rd to 8th grade students were not proficient
or partially proficient in English on the 2018-19 New York standardized tests compared to 55% testing
the same statewide and 64% were below proficiency in math compared to 53% statewide. COVID-19
is expected to widen these pre-existing inequalities in education and be particularly damaging to low-
income and students of color. In 2018-19, over 87% of our students were economically disadvantaged
and 97% represented minority populations. [New York City Department of Education (NYCDOE),
nycenet.edu] As an education-focused nonprofit organization, our mission is to help school districts
improve graduation rates and advance racial equity in education by bridging the gap between what
their students need and what most urban schools serving low-income populations are designed and
resourced to provide. Through our holistic model, Whole School Whole Child (WSWC), AmeriCorps
members serve in schools where they provide academic and social-emotional support to students,
serve as an essential resource in the classroom, and contribute to a positive school culture, climate,
and community. The social-emotional component of our program and our focus on creating positive
learning environments are particularly critical at this time in supporting students with overcoming
learning loss due to school closures (Durlak et al., 2011; Taylor et al., 2017). INTERVENTION:
CYNY requests to field 255 full-time members, who will be deployed in teams to their assigned school
-
DRAFT
Page 3
For Official Use Only
Narratives
for the entire academic year. The tiered WSWC structure establishes a preventative and intervention-
based framework through which members 1) as Student Success Coaches, are matched with students
to deliver targeted mentoring in academics, attendance, and/or social emotional development, 2)
serve in classrooms daily where they support classroom routines, positive behavior, and teachers'
instruction, 3) support before and after school programming, which includes homework assistance,
and, 4) work with school leaders to create positive school climate. Student Success Coaches provide
asset-based mentoring, including coaching on attendance, to their assigned students throughout the
day and provide 40 to 90 minutes of ELA or math support using evidence-based practices to each
mentee each week. Enrichment programming and positive climate activities are selected by the
individual school and can include: Morning Greeting, breakfast and homeroom engagement, recess
and transition time support, parent and family engagement activities (e.g. family nights, phone calls
home, and open houses), enrichment programming (e.g. talent shows, afterschool programs, field
trips, sports, student clubs, etc.), and, events that engage all students (e.g. assemblies, Poetry Month,
Spelling Bees) and/or recognize positive behaviors, such as attendance and academics awards, VIP
lunch clubs, and the award store. If needed, as a result of COVID, many of our services have been
adapted for delivery through in-person or virtual platforms. All services are selected in coordination
with the individual school so that they do not duplicate or displace existing school services or staff.
Periodic activities occur outside of the member regular schedule such as evening and weekend events
at our schools, and occasional CYNY staff approved additional service opportunities with city and
community-based organizations, such as our school Saturday tutoring programs and food banking in
our communities. CYNY has three types of member roles: 1) First Year Member, 2) Second Year
Member, and 3) Team Leader. All members support school-based activities as described above. Second
Year and Team Leaders serve as peer mentors and team coordinators and participate in advanced
level professional development and equity programs. In addition, Team Leaders provide logistical
leadership for our afterschool programs, lead team meetings and peer trainings, and represent the
team at CYNY and school meetings. VOLUNTEERS: Members will engage 200 volunteers through
physical service activities in our school communities and on national service days. All activities are
approved by CYNY staff to ensure that they do not include prohibited activities. THEORY OF
CHANGE: As the holistic WSWC framework was developed using evidence informed practices, our
theory of change was developed through external evaluation (Brett Consulting Group, 2008, 2009,
and 2010) and input from thought leaders in education. The WSWC theory of change is that when
near peers, our members, mentor students using social-emotional informed practices, students will
-
DRAFT
Page 4
For Official Use Only
Narratives
develop self-identity, a sense of agency, and academic and life skills, such as classroom participation,
social skills, problem solving, and critical thinking, which will make the student more likely to enter
tenth grade prepared to successfully graduate from high school and navigate college and career
(Learner, 2004, Balfanz & Byrnes, 2013). EXPECTED OUTCOMES: On an annual basis we expect to
see meaningful short-term improvement related to literacy and math and gains in attendance and
social-emotional development. Over multi-year exposure to our program, we expect to change the
mindset of students so that they are more motivated to learn and succeed academically and attend
school consistently. Our long-term goal is that by serving alongside students from elementary to high
school, students will graduate at higher rates, be better prepared, and have a more positive viewpoint
of their post-secondary success and career. RATIONALE FOR USING AMERICORPS: The full-time
aspect AmeriCorps program provides NYCDOE schools a cost-effective approach that offers wrap
around services to students and meets the district commitment of creating and supporting learning
environments that reflect the diversity of NYC. For example, we support the NYC Department of
Youth and Community Development (DYCD) in delivering the SONYC afterschool program to
middle school students. Through the AmeriCorps program, we can recruit members who are closer in
age and experiences to students than teachers and have members support school efforts to be
culturally responsive, while teaching students how to resolve conflict and build positive relationships.
CONTRIBUTIONS TO EXISTING EFFORTS: Since 2014, central to the Mayor and Chancellor's
vision for education in New York is the development of Community Schools, which provide wrap
around services to students beyond the traditional school offerings, including social-emotional and
mental health support, parent empowerment, and extended learning time. At the heart of the
Community Schools holistic model is a partnership between a community-based organization (CBO)
and school leadership. CYNY currently partners with eight Community Schools and is the lead CBO
partner at three of the eight schools. For 2021-22, we have applied to be the lead partner in up to
twenty Community Schools. Expansion of our program into more Community Schools would support
NYCDOE's efforts to reach students with critical needs that the traditional school structure and
families are not equipped to provide. PERFORMANCE MEASURE SELECTION: CYNY is submitting
two aligned performance measures that corresponds to our academic mentoring in English
(ED1A/ED5) and math (ED1A/ED5). While we provide social-emotional and attendance mentoring,
we are not providing aligned measures for these activities at this time due to our on-going learning
around objective outcome instruments in these areas. The selected measures reflect our significant
program activities and the role of the Student Success Coach, in which members support students
-
DRAFT
Page 5
For Official Use Only
Narratives
with their social-emotional development, academic outcomes, and school engagement on a daily
basis. RATIONALE FOR TARGETS: The proposed targets are based on prior year data, projected class
schedule, and member capacity. Our AmeriCorps members have a "Focus List" of up to thirteen
students who they mentor daily in one or more indicators, e.g. English, math, etc. The improvement
rates and definitions are realistic and ambitious in terms of the academic starting point of our students
and factors outside of our control, e.g. as a result of COVID-19 school closures and inconsistent
distance learning, low-income students may fall 12.4 months behind in catching up compared to a
projected national seven-month average for all students (McKinsey & Company, June 2020).
2. EVIDENCE BASE: Our program has strong evidence and CYNY is included in three of the four
studies which include City Year on the AmeriCorps Evidence Exchange (Policy Studies Associates,
2014; Abt Associates, 2016, MDRC, 2016, Chapin Hall, 2017). For this application, we are submitting
two evaluation reports. The findings of the two studies align with the outcomes in our logic model and
the tiered structure of our program design that includes both prevention and targeted mentoring
around academics, attendance, and social-emotional development. CYNY data is included in both
studies. STUDY #1: "Analysis of the Impacts of City Year's Whole School Whole Child Model on
Partner Schools' Performance", Policy Studies Association (PSA), 2015. STUDY DESIGN: This
evaluation is a Quasi-Experimental Design (QED) study of the City Year program in multiple
locations over multiple years. The study assessed publicly accessible testing data in English and math
over three years, 2012 to 2014. The analyses focused on approximately 600 schools in 22 cities and
included 150 schools that partnered with City Year and 460 comparison schools that did not. This
study was placed in the strong evidence tier by CNCS for the 2019-20 and 2020-21 program years.
KEY FINDINGS: This study included multiple statistically significant positive findings (e.g. p
-
DRAFT
Page 6
For Official Use Only
Narratives
skills and multiple measures of education outcomes (attendance, course grades, and achievement
tests). City Year provided ECG with data collected for 38,131 students (grades 3-10) who participated
in our programs. Data included student-level data on literacy and math assessments, course grades,
attendance rates, social-emotional skills observed or self-reported, student demographics, and hours of
intervention received. KEY FINDINGS: Analysis found that there was a statistically significant
relationship between students' social-emotional skills and their academic outcomes (effect size range
0.08 to 0.42), which is estimated to be the equivalent of an entire school year of academic
achievement in English or math. The study also found that the more hours students spent with City
Year members, the more likely they were to improve social-emotional levels (effect size 0.06 - 0.08;
2-3 percentile points) and English or math (effect size 0.08-0.09; 2-4 months of learning), and, the
less likely they were to be off-track in English class (odds-ratio = 0.58) and math class (odds-ratio =
0.66). These findings validate that the WSWC holistic approach has a positive impact on student
social-emotional development and academic success. SAME INTERVENTION: The beneficiary
populations in both evaluations are similar to our current school partner populations in terms of state
standardized test performance data, student enrollment, grade ranges served, and student
demographic data including race and low-income status. The member population continues to be
young adults (under 26 years old) and diverse across racial, ethnic, and education backgrounds. The
overall structure, design, and dosage of WSWC is the same. City Year conducts an annual
implementation assessment (survey completed by the supervisor of each school team) to ensure that
the model is consistent. OTHER EVIDENCE: Between 2017 and 2019, CYNY served, on average each
year, 908 students in ELA, 803 in math, and 522 in attendance, of which 36% improved in ELA, 51%
in math, and 30% in attendance. In 2020, we did not receive end of year data. CYNY was included in
a randomized control trial of the Diplomas Now partnership which indicated that students receiving
WSWC services are more likely to report positive relationships with adults (p=0.011) (MDRC, June
2016). Within our schools, a strong majority of teachers and administrative staff agreed or strongly
agreed that CYNY members help improve learning environments, improve student engagement and
participation in their academic learning, and positively impact students' overall academic
performance. In addition, schools reported that our AmeriCorps members serve as role models for
students who lack other positive role models in their lives (PSA, 2014).
3.NOTICE PRIORITY: CYNY aligns with evidence-based programs under the Education area
funding priority (see Evidence Base). Under the COVID priority area, our focus on social emotional
and relationship development will support students in re-engaging with their academic experience,
-
DRAFT
Page 7
For Official Use Only
Narratives
thereby helping them recover from learning losses due to COVID (Houghton Mifflin Harcourt, 2020).
4.MEMBER EXPERIENCE: SKILL DEVELOPMENT: All City Year sites share a common set of
mission-aligned beliefs and values that shape the member experience. CYNY members participate in a
six-week gradual release basic training academy hosted at Hunter College, a mid-year summit, and
learning and development workshops that foster service and professional skills, build community, and
allow members reflect on their experience. We use the GROW coaching strategy designed for young
adults to help members structure their professional aspirations and process their service experience.
Our alumni report that the most valuable skills gained were in team collaboration and leadership by
managing a group of their peers through team Coordinator Roles, group planning for afterschool
programs, participating in 360 Degree feedback protocols and Behind Closed Doors workshops, and
providing thoughtful feedback and input on our program. All members participate in our Leadership
After City Year (LACY) program, which provides workshops that advance members' professional and
personal development, e.g. resume writing, financial literacy, personal and professional brand, and
alumni and community leader panels. Through our member development programs, we are able to
engage community partners such as iMentor, Teach for America, NYC Teaching Fellows, NYC
Service, Deloitte, and NY Life. RECRUITMENT & INCLUSION: CYNY seeks applicants from
different backgrounds, abilities, education levels, geographic location, and race/ethnicity. We have
two recruiters focused on recruitment from CYNY's communities and leverage City Year's national
recruitment partnerships and our alumni board to create a diverse corps that reflects the demographic
communities we serve. Applicants receive individualized support throughout the application process
and are engaged through team shadow days, info sessions, and invitations to CYNY events such as
volunteer events and graduation. To create an inclusive community, members serve on diverse teams;
wear uniforms; utilize standardized meeting protocols that promote inclusion; participate in CYNY
and City Year network-wide affinity group discussions, equity and diversity learning activities, and
community meetings through in-person or on virtual platforms; and, have small-group facilitated
discussions that reflect on individual identity through our Idealist's Journey curriculum. Members
have access to mental health supports through our employee assistance program, Talkspace
partnership, and monthly sessions provided by New York University. Feedback from the corps is
solicited through our corps council, surveys, and roundtables with CYNY staff. Two members serve
on the City Year network-wide Dean's Council, which informs City Year on issues facing the corps.
CYNY members will be connected to other NY AmeriCorps programs through participation in the NY
Service Summit and national service day events.
-
DRAFT
Page 8
For Official Use Only
Narratives
Organizational Capability
1.ORGANIZATIONAL BACKGROUND AND STAFFING: City Year New York is an operating site
within the national 501c3 non-profit organization City Year, Inc. Founded in 1988, City Year is
managed by a National Board of Trustees and each operating site, such as CYNY, is led by an
Executive Director. To gain community input and support, CYNY maintains five advisory boards:
leadership, associate, investment, women's, and alumni. In 2021-22, the CYNY program staffing
model provides sound oversight and is split into the functional areas of service delivery, partnerships,
learning and development, data and evaluation, member experience, and human resources and
operations. The Managing Director of Impact (MDI) oversees thirty-eight Impact staff and is
responsible for overall program leadership and management and school district and AmeriCorps
relations. Our twenty school teams are split into two initiatives, Community Schools and SONYC
Schools, each led by a Senior Impact Director (SID). Within each initiative, an Impact Director (4)
manages four to six school relationships and service delivery and quality. Each school campus is
supervised by an Impact Manager (see Member Supervision). Four Community School Directors will
lead partnership efforts within their portfolio of schools. Five instructional staff support member
training and development and provide on-site coaching on service delivery to staff and corps. Two
staff oversee our data collection and evaluation efforts. One staff member supports members with
navigating their service experience from enrollment to exit and supports the LACY program and
transition to alumni. The Managing Director of People and Operations leads five staff in managing all
corps member and staff human resources, office operations, and equity programs. City Year
Headquarters (CYHQ) supports CYNY by providing centralized services for criminal history checks,
member enrollments/exits, finance and payroll operations, information technology, and member
recruitment. The Managing Director of Impact has over 15 years' experience leading the CYNY's
AmeriCorps program, and, over the past decade, CYNY has managed over $37M in AmeriCorps
awards, filled 3,147 member positions and 99%+ of awarded slots, retained 88% of our members, and,
submitted >97% of enrollments and exits on time. Of this, over $21M was awarded through New
Yorkers Volunteer through State Competitive funding.
2.COMPLIANCE AND ACCOUNTABILITY: Our processes are designed to meet the requirements of
an AmeriCorps program and provide for on-going monitoring and oversight of member file
documents, criminal history checks, and service activities. To prevent non-compliance, practices
within our enterprise solution, Workday, are designed to meet federal grant regulations, such as
separation of funds and time and effort certifications. We use Truescreen and Fieldprint to conduct
-
DRAFT
Page 9
For Official Use Only
Narratives
criminal history checks. To detect reporting errors, CYHQ staff conduct internal audits to ensure
completion of member file documents, including criminal history checks, and at least two finance
staff review AmeriCorps invoices and financial reports. Each school completes a School Partnership
Agreement, which includes the AmeriCorps prohibited activities, non-displacement, and non-
duplication requirements. To monitor service activities and our partnership relationships on a day to
day basis, we place Impact Managers in our schools to supervise the team. Impact Directors and
Instructional Coaches visit the schools bi-weekly and we have monthly partnership check-ins with
school administration. Each December and March, the Impact Manager and Impact Director for the
school complete a service partner rubric to assess the state of the partnership. Service implementation
checklists and our observation and coaching tracker assist us with identifying school needs so that we
can direct resources and support to each team. Externally, the DYCD and the NYS Department of
Health and Mental Health tests adherence to city mandated safety protocols for our afterschool
programs through periodic unannounced visits. For financial accountability, our National Board of
Trustees maintains a finance and audit committee that meets quarterly and oversees our annual
Single Audit and approves our national internal controls and financial policies. Under our National
Direct grant, CYNY has been participating in an external evaluation through MDRC and AIR that
has an approved Alternative Evaluation Approach (AEA). Per the AEA guidance, we are submitting
an interim evaluation report.
3.CULTURE THAT VALUES LEARNING: CYNY strives to create a data informed culture, which
shapes our program development, staff and corps learning and engagement, and strategic direction.
We administer engagement and satisfaction surveys to corps members, staff, and school partner staff
at least twice annually, maintain an evaluation partnership with Glass Frog Solutions, and participate
in City Year's national evaluation and learning agenda. Analysis of the surveys and our student-level
data is regularly shared and discussed with staff, corps, and our external partners through meetings,
retreats, and newsletters. As part of our partnership with the NYCDOE, we have conducted an annual
independent evaluation of our service for over a decade. Most recently Glass Frog has completed an
independent analysis our Whole School Whole Child program for school years 2018 and 2019. These
evaluations allow us to dig into our performance measure data and understand variation in service
and impact across schools, student peer groups, and types of services received. We use the evaluation
reports to refine our service model and advance our school partnerships to maximize our impact
within the schoolhouse. To build learning communities within our corps, we split them into learning
cohorts of not more than 40 corps members who share similar service experience, e.g. same grade
-
DRAFT
Page 10
For Official Use Only
Narratives
Budget/Cost Effectiveness
level but from different schools. Cohorts meet at least monthly for up to 90 minutes to engage in
subject-specific learning activities, e.g. third grade social-emotional strategies, that incorporates active
engagement and application exercises. The whole corps meets in-person or virtually for Community
Meetings, which are used to educate our community about the state of our service and issues of
educational equity and share life experiences through individual testimonials. In addition to our local
learning and evaluation agenda, CYNY staff and corps participate in the City Year network-wide
communities of practice, for example, weekly or monthly calls depending on role and subject and
national and regional retreats for staff and corps that focus on data, service implementation, equity
and inclusion, and/or school relationship development. Communities of practice are designed to create
learning networks across our staff and corps that foster continuous improvement.
4.MEMBER SUPERVISION: SUFFICIENT GUIDANCE & SUPPORT: Impact Manager (IM)
supervise one school campus and provide day-to-day supervision of 9 to 21 AmeriCorps members. IMs
are in schools four to five days a week where they coach our members through their professional
development by helping members set goals, evaluating their performance, and providing them
feedback on their service. IMs have monthly one-on-ones with each member of team during which
they monitor for prohibited activities and address any school, service, or personal issues the members
may have. IMs are responsible for overseeing the member calendar, verifying member service hours
and approving timesheets, and reporting any member issues to their supervisor, CYHQ, and/or the
Office of the Inspector General as appropriate. TRAINING & PREPARATION: The first six weeks of
the program year are used for training and planning for the upcoming year. Training content includes
AmeriCorps policies, our service model and goals, NYCDOE partnerships and expectations, student
progress monitoring practices, the Clover youth development model, leading with an equity mindset,
coaching skills, crucial conversations, collaborative goal setting, and City Year policies and
information systems. The Learning and Instruction team provides training in curriculum delivery and
mentoring practices. School partners provide a start of year training that includes an introduction to
school faculty, school priorities and protocols, and teacher classroom orientations. All IMs complete
professional development plans and participate in twice monthly training opportunities. IMs have
day-to-day support from their Impact Director, Instructional Coach, and Community School Director
and have peer support through monthly Impact team meetings, national or regional training events
led by CYHQ, and twice annual Impact team retreats.
See budget.
-
DRAFT
Page 11
For Official Use Only
Narratives
Evaluation Summary or Plan
CITY YEAR NEW YORK 2021 EXTERNAL EVALUATION PLAN
EVALUATION PLAN SUMMARY:
EVALUATOR: MDRC and American Institutes for Research
STUDY FOCUS: Impact of the Whole School Whole Child on student and school-wide social-
emotional and academic outcomes
STUDY TYPE: Randomized control trial and quasi-experimental design study
GRANT YEARS COVERED: 2018-19 and 2019-20
APPLICABLE RECOMPETING APPLICATIONS:
* City Year Jacksonville: 18ACHFL0010001 (21AC230127)
* City Year Los Angeles: 18ACHCA0010003 (21AC230042)
* City Year New York: New/Recompete (21ES231071)
* City Year Orlando: 18ACHFL0010002 (21AC230126)
* City Year San Antonio: 18ACHTX0010002 (21AC230077)
PRIOR AEA REQUEST: Requested on the basis of timing. Following AmeriCorps AEA guidance, an
Implementation Report of work conducted to date has been submitted with the application.
***********************************
EVALUATION PLAN NARRATIVE:
PROGRAM BACKGROUND: The City Year (CY) Whole School Whole Child (WSWC) service model
is a two-tiered model providing universal and targeted academic and socio-emotional mentoring as
well as attendance-monitoring services. City Year recruits and places teams of full-time AmeriCorps
members (ACMs) in schools to serve as Student Success Coaches and deliver the WSWC model. ACMs
-
DRAFT
Page 12
For Official Use Only
Narratives
provide universal (Tier 1) services to all students by assisting teachers in the classroom; supporting
non-instructional time outside the classroom; and organizing school events, community service
projects, and afterschool programs. ACMs also provide individualized (Tier 2) services -individualized
mentoring that develops skills to support English, math, attendance, and social-emotional
development - to students who exhibit early warning signs for not graduating on time.
THEORY OF CHANGE: The WSWC theory of change is that when near peers, our members, mentor
students using social-emotional informed practices, students will develop self-identity, a sense of
agency, and academic and life skills, such as classroom participation, social skills, problem solving,
and critical thinking, which will make the student more likely to enter tenth grade prepared to
successfully graduate from high school and navigate college and career (Lerner, 2004). Further, when
students who are most at-risk receive individualized services in a supportive school environment,
other students improve overall attendance, behavior and course performance due to decreased
distractions for students and teachers (Balfanz & Byrnes, 2013).
OUTCOMES OF INTEREST: As part of the planned evaluation City Year has developed two logic
models to align with universal (Tier 1) services and targeted (Tier 2) services. The Tier 1 logic model
outlines whole school services and connects these program components with attended outcomes. For
example, connecting whole class support in reading and mathematics classes to schoolwide
performance on state standardized assessments. The Tier 2 model utilizes the same inputs and
activities but adds individual mentoring and student progress monitoring, which are then connected to
outcomes at the individual student level. For example, the activity for attendance is individual
coaching and mentoring versus school-wide programming and the outcome is the number of students
who improve attendance versus improvement in the schoolwide attendance rate. The long-term
outcome for all activities is to increase high school graduation rates and have more students exit high
school prepared for post-secondary success. Outcomes of interest include: improved schoolwide and
individual attendance; improved schoolwide and individual course grades and standardized test scores
in reading and math, improved individual goal mastery and growth mindset, improved individual self-
management and emotional intelligence, and improved school climate, specifically, student-adult and
student-student relationships.
RESEARCH QUESTIONS: The study of the WSWC model, both Tier 1 and Tier 2 services, will
address three impact questions: (1) Do students in City Year middle schools have better academic,
behavioral, and school engagement outcomes and experience more positive school climate than
students in comparison schools?; (2) Do the effects vary by student subgroup (age, gender, race or
-
DRAFT
Page 13
For Official Use Only
Narratives
ethnicity)?; and (3) Do the effects increase over time, as schools implement the WSWC model for
more years? The study will address two implementation questions: (1) To what extent are services
provided with fidelity? and (2) How do variations in implementation fidelity explain variations in the
observed effects? A cost analysis will address this research question: What are the monetary costs of
implementing the WSWC model in urban middle schools?
RESEARCH & EVALUATION DESIGN: In February 2017, the Institute of Education Sciences (IES)
Education Research Grants Program awarded a grant for a multi-year efficacy study of our Whole
School Whole Child model in high need urban middle schools to the American Institutes of Research
(AIR), MDRC, and Stephanie Jones (Harvard University). The study will evaluate the routine
conditions of WSWC and test the efficacy of City Year's WSWC model and the added value of targeted
(Tier 2) services. The study will be conducted in five districts serving large urban areas: Los Angeles
Unified School District (CA), New York City Public Schools (NY), Duval County Public Schools (FL),
Orange County Public Schools (FL), and the San Antonio Independent School District (TX). The study
includes a student-level randomized controlled trial (RCT) to assess the effects of Tier 2 services on
students after 1- and 2- years' participation, a comparative interrupted time series (CITS) design to
assess the effect of the WSWC model, and an implementation analysis to examine the costs of
implementation. The RCT includes Tier 2 student level data from the evaluation years (2018-19 -
2019-20) and the CITS includes whole school data from 2007-08 to 2018-19. The RCT was originally
planned to include two cohorts of students where cohort 1 received services in 2018-19 and 2019-20
and cohort 2 was to receive services 2019-20 and 2020-21. Due to COVID-19, the investigators
eliminated the second cohort. The CITS study was originally planned to extend through 2020-21,
however, due to COVID-19, the 2019-20 and 2020-21 years have been eliminated. The RCT is being
conducted by AIR and the CITS and implementation analysis is being conducted by MDRC.
RATIONALE: In addition to the highly rigorous RCT framework for Tier 2 services, the CITS design
is more rigorous than most other quasi-experimental design (QED) studies because it combines
longitudinal data and a matched comparison group. CITS designs have been used to evaluate federal
policies like No Child Left Behind (Dee & Jacob, 2011; Wong, et al, 2015) and grade-level reform
models (Somers & Garcia, 2016). Several studies have shown that CITS can produce statistically
similar results to an RCT under certain conditions (Fretheim et al., 2013; Schneeweiss, et al 2004;
Somers et al., 2013; St. Clair, et al, 2014), which are expected to be met in this study.
STRENGTH AND LIMITATIONS: Possible biases to the comparison group's baseline trends could
occur through policy shock or instructional reform, which may affect either the City Year group or
-
DRAFT
Page 14
For Official Use Only
Narratives
control schools; and / or comparison schools may experience different demographic shifts in student
populations. Due to COVID-19, several aspects of our IES evaluation have been impacted. For
example, the evaluators will be excluding the second RCT cohort from the study as services were
significantly disrupted due to school closures.
SAMPLING: The sample includes students identified in early sixth grade as needing support relative to
literacy, math, attendance, and behavior; in 22 City Year schools and approximately 30 matched
comparison schools in five urban school districts (Los Angeles, New York, Jacksonville, Orlando, and
San Antonio). The CITS study will include data from school years 2007-2008 to 2018-19 representing
nearly 300,000 students. The RCT study includes approximately 3,300 students in grades 6-8 enrolled
in these schools during the evaluation years.
DATA SOURCES: MEASUREMENT TOOLS AND DATA COLLECTION: Measurement tools will
include our standard suite of student pre/post assessments (determined by school and district), report
card grades, and observational data. Student pre- (fall) and post- (spring) surveys, as well as all
relevant student outcome and intervention data, will be collected by City Year throughout the course
of the study and shared with the researchers to run analyses and report out results. Whole school data
will be collected through public resources (state department of education websites) by the researchers.
To accommodate changes in school assessment schedules and testing due to COVID-19, for 2019-20,
mid-year data may be used instead of end of year. IES is aware that study years have changed as a
result of COVID-19.
ANALYSIS PLAN: Both the RCT and CITS studies will use hierarchical linear modeling to account for
clustering of data. The CITS study will be used to evaluate the effects of the full WSWC model. Both
studies will conduct student subgroup analyses and will document implementation fidelity. Both
studies will measure student outcomes in three domains: academic achievement; social, emotional,
and behavioral; and school engagement.
EVALUATOR QUALIFICATIONS:
American Institutes for Research: AIR has earned an international reputation for advancing
educational science and has evaluated programs addressing middle school improvement, dropout
prevention, and high needs youth. AIR has conducted over six RCTs, five QEDs, two CITS, and 15
mixed-methods studies that address social-emotional development in secondary schools.
MDRC: MDRC is an education and social policy research organization with a 40+-year history of
conducting large-scale demonstrations and evaluations. MDRC is widely considered a leading
institution in rigorous evaluations of school reforms, has conducted numerous large-scale
-
DRAFT
Page 15
For Official Use Only
Narratives
experimental and quasi-experimental evaluations within schools, and has worked with hundreds of
middle and secondary schools.
The seven team members included in this proposal have expertise in the areas necessary for making
this project a success. David Osher, Ph.D., Vice President at AIR, will serve as the Principal
Investigator (PI) with Marie-Andrée Somers, Ph.D., Senior Research Associate at MDRC, as a Co-PI.
TIMELINE AND BUDGET: The IES award period is 9/1/2017-12/31/2021. The planning year was
held in 2017-18. The evaluation years span 2018-19 to 2020-2021. The report will be written during
the 2021-2022 program year. Reports are scheduled: interim report, Year 3, final report, Year 4, and a
manuscript for peer review, Year 5. The IES grant award budget is $3.2 million. Timeline, report
schedule, and budget subject to change per MDRC and AIR negotiation with IES.
*****************************************
REQUEST FOR ALTERNATIVE EVALUATION APPROACH:
City Year New York has operated through City Year's National Direct grant over the previous grant
cycle. Under our 2018 National Direct Recompete Application, we submitted an Alternative
Evaluation Approach (AEA) on the basis of timing for the IES evaluation as it is a five-year study. As
this study will continue into the next grant cycle, we are requesting to renew the AEA on the basis of
timing. In addition, City Year New York has already completed a prior experimental impact
evaluation that demonstrates evidence of effectiveness in one or more key desired outcomes that are
aligned with our theory of change. Evaluation constraints, rigor, and contribution to City Year's
learning base are as described:
A) CONSTRAINTS: The constraints relate to funding, capacity, and redundancy. City Year invests in
national studies to conduct experimental and rigorous evaluations of the program design as a whole
and localized studies with lower rigor to test specific components of the program design. Over the past
three years, City Year, Inc. has invested in six multi-site external evaluations through the Everyone
Graduates Center, Policy Studies Associates, Research Triangle Institute, Deloitte Consulting, and
MDRC and AIR. Of these, one has been completed and the remaining five are scheduled to be fully
completed by the summer of 2022. Our organizational staffing and financial capacity to add another
evaluation at this time are limited. For the proposed IES study, the lead investigators, MDRC and
AIR, established the five-year time horizon and set the schedule for reports. In addition, following
-
DRAFT
Page 16
For Official Use Only
Narratives
standard practices during randomized control trials, MDRC and AIR will have locked data sets
meaning that we cannot share the data for other research evaluations during the five-year study
period, which limits both our ability to receive reports and to conduct other evaluations using the
same data.
B) RIGOR: The national evaluation approach provides us the opportunity to secure grants through
the U.S. Department of Education that support high quality and rigorous evaluations, such as the
multi-year randomized control trial described in the evaluation plan, that we otherwise would not
have the resources or capacity to implement. A randomized control trial is the highest level of rigor.
By comparing the characteristics of each evaluation study's cohort of City Year schools and districts
with those served by City Year but not participating in the study, we infer that the conditions our
members operate in will be sufficiently comparable regardless of direct study involvement. We can
also reasonably conclude that program impact will be consistent regardless of location. This
conclusion has been supported by evaluator MDRC's Principal Investigator in reference to our
participation in the national evaluation of Diplomas Now (i3).
C) CONTRIBUTION TO EVIDENCE BASE: All City Year evaluations, local or national, contribute to
the organizational body of evidence and inform on our learning agenda. City Year leverages research
to inform practice and engages in evaluation to better understand our impact and continuously
improve the services we provide to AmeriCorps members and students. Each study, including the
proposed study, is used to inform on our learning agenda and our next study's research questions and
test assumptions. For example, future areas of inquiry include the role of learning environments and
relationships in shaping student, educator, and AmeriCorps member experiences, the role of
communities in supporting school outcomes, understanding effective integrated social, emotional, and
academic strategies, and how service impacts member future experiences. The IES fills a gap in our
evidence in that City Year does not currently have experimental research evidence on the effectiveness
of WSWC as a whole. The IES efficacy study evaluates the effects of the WSWC model and the added
value of Tier 2 services. The proposed project is important to address the need, population, evaluation
design, timing of outcomes, and the social and behavioral context for academic learning.
TIMING AEA:
As included in our Evaluation Plan, the IES study is anticipated to take 5 years with a report being
issued in 2021 and a peer-reviewed manuscript to be issued in 2022. The final report will be available
-
DRAFT
Page 17
For Official Use Only
Narratives
at our next recompete application in fall 2023. As noted, we do not control the timeline for this
evaluation and have limitations on using data for the study operating sites until the evaluation is
complete.
PRIOR EVALUTION AEA:
City Year has a robust evaluation and research program that ensures that all sites are included in
studies that are rigorous and contribute to our evidence base. City Year New York has participated in
a prior randomized control trial and two quasi-experimental design studies as well as non-
experimental studies. A selection of studies includes:
EVERYONE GRADUATES CENTER: "Connecting Social-Emotional Development, Academic
Achievement, and On-Track Outcomes: A Multi-District Study of Grades 3 to 9 Students Supported by
City Year", 2020.
* Study Type: Non-Experimental Mixed Methods
* Evidence of Effectiveness: This study demonstrated that for students who participate in our program
there are statistically significant and consistent relationships between students' social-emotional skills
and their academic outcomes across grade levels, and that, the more hours of support a student
receives from an AmeriCorps member, the higher the student outcomes in the related subject (ELA or
math), attendance, and social-emotional skills.
GLASS FROG SOLUTIONS, "Examining the Factors Associated with Student Growth at City Year
New York", 2019
* Study Type: Non-Experimental Quantitative Analysis
* Evidence of Effectiveness: This study was designed to identify best practices to inform decision-
making within each of our programmatic areas. The findings show that CYNY has year over year
consistency in outcomes across indicator areas and that certain subgroups of students and assessment
tools have consistently higher outcomes than others. The study included data from program years
2015-16 to 2018-19.
MDRC, "Addressing Early Warning Indicators: Interim Impact Findings from the Investing in
Innovation (i3) Evaluation of Diplomas Now", 2016.
-
DRAFT
Page 18
For Official Use Only
Narratives
Amendment Justification
Clarification Summary
Continuation Changes
* Study Type: Randomized Control Trial
* Evidence of Effectiveness: This study demonstrated that our program had a statistically significant
impact on reducing the percentage of sixth and ninth graders demonstrating early warning indicators
and on reducing chronic absenteeism.
* Evidence Tier: Under the 2018 grant competition, this study was assessed by CNCS to be in the
Strong evidence tier and is posted on the Evidence Exchange.
POLICY STUDY ASSOCIATES, "Analysis of the Impacts of City Year's Whole School Whole Child
Model on Partner Schools' Performance", 2015.
* Study Type: Quasi-Experimental Design
* Evidence of Effectiveness: This evaluation demonstrated that our program had statistically
significant impact on improving whole school performance on literacy and math assessments.
* Evidence Tier: Under the 2019 and 2020 grant competitions, this study was assessed by CNCS at the
Strong evidence tier.
POLICY STUDIES ASSOCIATES, "Year 4 Evaluation of City Year New York's Whole School Whole
Child Initiative", 2014
* Study Type: Non-Experiment and Quasi-Experimental Design
* Evidence of Effectiveness: This study focused on implementation and identifying areas for
improvement. In addition, it provided additional analysis of City Year performance measure data and
City Year administered surveys. The analysis found that schools believe that City Year has an impact
on student academic outcomes and academic engagement.
* Evidence Tier: This evaluation was assessed at the preliminary evidence tier and is posted on the
Evidence Exchange.
Not applicable.
Not applicable.
Not applicable.
-
DRAFT
Page 19
For Official Use Only
Grant Characteristics
-
DRAFT
Page 20
For Official Use Only
Education
Focus Area
100%
% MSYsK-12 Success 100%
K-12 Success
Objectives
255.00
No of MSYs
255
No of Members
Table1: MSYs by Focus Areas Table2: MSYs by Objectives
Table4: No of MSY and Members by Objective
NPM Applicant 69% 0%% MSYs
Table3: %MSYs by NPM vs.Applicant vs. Not in ANYNot in ANY 31%
Primary Focus Area:
Secondary Focus Area:
Education
Education
Primary Intervention:
Secondary Intervention:
Mentoring
Other Classroom Support
Performance Measures
Objectives %MSYs
255.00Total 255
-
DRAFT
Page 21
For Official Use Only
Problem Statement:
Selected Interventions:
MentoringOther Classroom Support
Academic Services in English Language ArtsPerformance Measure:
Education K-12 SuccessFocus Area:
Objective:No of Members:
93.00 148No of MSY's:
Describe Interventions:
DESIGN: City Year New York members are embedded in 3rd to 10th grade classrooms for English Language Arts where they serve as Student Success Coaches and provide focused academic mentoring in literacy and whole class support. In classrooms, members support the whole class in following instructional routines and provide small-group focused support using NYCDOE literacy protocols. Mentoring includes improving durable literacy skills and supports development of good study habits, such as note taking, collaboration, task prioritization, and homework completion. Member activities improve student on-task behavior and limit peer distraction while addressing student motivation and academic engagement, such as in-class assignment completion and participation. Members also model good academic behaviors and reinforce the concepts of a collaborative classroom climate. Outside of the classroom, members develop a culture of literacy through literacy-based events and enrichment programming. DOSAGE: All students in the classroom are substantially engaged through full-time placement of members in the classroom for the academic year. Classes are 45-90 minutes per class five days per week. Focused literacymentoring occurs in 30 to 45-minute sessions 2x per week. Each student included under ED1A is expected toreceive at least 60 minutes of individualized support per week. Members may also provide literacy support to
In 2021-22, City Year New York anticipates partnering with 20 elementary, middle and K-8/6-12 schools (5 are TBD) within the neighborhoods of Queens, Harlem, Brooklyn and the Bronx, in order to improve student outcomes that lead to improved graduation rates. In 2018-19, City Year’s proposed partner schools enrolled nearly 7,000 students, the majority of whom represent minority (avg. 95%) and economically disadvantaged (avg.87.4%) backgrounds (2018-19 School Performance Dashboard, NYC DOE). Annual state testing shows that low-income and minority students perform lower on standardized testing for literacy than their more affluent and white peers and are at risk for not graduating high school on time. For example, 16% of students who are not reading at proficiency by the end of 3rd grade will not graduate high school on time, however, among low-income students in general, 26% will not graduate on time and among low-income students who are also livingin neighborhoods of concentrated poverty, 35% will not graduate on time (Hernandez, 2012). On the 2018-19 New York State Testing Program (NYSTP), 61% (3,008/4,949) of students tested in 3rd-8th grades at City Yearpartner schools scored below proficiency (Partially/Not Proficient) on the English portion of the exam. In comparison, 55% (539,119/987,398) of 3rd-8th grade students statewide, and of these, 53% (206,434/392,460) of students citywide tested below proficiency in ELA this same year (2018-19 School Performance Dashboard, NYC DOE). The city’s overall graduation rate has increased to a high of 77.3% in 2019, but is still significantly lower than the state’s graduation rate of 83.4% (2019 NY State Graduation Rate Data). The Early Warning Indicators predict that students who are below proficiency or failing English are at high risk to drop out (Balfanz & Neild, 2006). Further, early intervention in English before the 9th grade is key to getting struggling students back on track for graduation (Corrin, et al, 2016).
-
DRAFT
Page 22
For Official Use Only
Problem Statement:
Selected Interventions:MentoringOther Classroom Support
Academic Services in English Language ArtsPerformance Measure:
Described Instrument: TITLE: CYSCHOOLHOUSE. cyschoolhouse is a secure cloud-based platform thatprovides an intervention management and tracking system to record student-level data.WHAT IS MEASURED: Student profile data includes name/identifying number, school, grade level, member assignment, which targeted interventions students receive, and participation. The Gradebook module of cyschoolhouse tracks individual student grades and an Assessment module tracks attendance data, other school based data, and assessments.HOW ADMINISTERED: cyschoolhouse is managed by City Year Headquarters. City Year New York staff enter student profile and school data with records provided by schools/district. Intervention data is entered by members using computer, tablet, or mobile phone application. RELIABILITY/VALIDITY: Only one record per student is created to ensure non-duplication, further City Year Headquarters cleans data to make sure that recordsare valid and not duplicated. Data is reviewed and analyzed by both City Year NewYork and Headquarters analytics staff. cyschoolhouse is FERPA compliant and was designed by Acumen Solutions for City Year in 2012 and vetted by a consortium of DC Charter schools. TARGET: Approximately 125 members will be assigned to ELA classrooms where they support all students in the classroom (25-30 students per member) and are assigned up to 10 Focus List students for additional intensive literacy supportabove standard strategies, e.g. outside of class. Targets are based on the subset of students assigned to members in English classrooms and reflect variations in school enrollment size, class schedule, and member capacity. Team Leaders do
ED1A: Number of individuals served
ED1A Output:
Tracking SystemMeasured By:
1072Target: Individuals
students outside of class time, e.g. before or after school. Once enrolled in our program, students are expectedto receive literacy services for the duration of the school year unless they withdraw from school or are assignedto another classroom. Combined class and focus time, students receive a minimum of 5 hours of literacy services per week.NYCDOE PROTOCOLS: Members will support students through use of the Levelled Literacy Intervention System and/or City Year’s research-based literacy protocols in fluency, vocabulary, and reading comprehension.
-
DRAFT
Page 23
For Official Use Only
Problem Statement:
Selected Interventions:MentoringOther Classroom Support
Academic Services in English Language ArtsPerformance Measure:
Described Instrument:
Described Instrument:
not carry assigned Focus List students but engage all Focus List students in the school.
TITLES: Outcome instruments are selected in partnership between City Year NewYork, the New York City Department of Education, and individual schools. Instruments: Fountas & Pinnell Reading Record (F&P) or English course grades.WHAT IS MEASURED: All instruments measure student level progress in English. Assessments measure skills and knowledge in the domains of literacy as applicable to the grade level: phonemic awareness, phonics, fluency, vocabulary, comprehension, and writing. Grades measure content mastery, homework completion, and participation. Improvement is defined as increasing by 1 reading level (F&P), or improving course grade or grade point average (depending on alphaor point grading system).HOW ADMINISTERED: F&P is a computer adaptive test administered by the school up to three times a year. Grades are issued by teachers at least four times a year. The first assessment or marking period grade or, when available, prior year end of year results, will serve as baseline/pre data and the last assessment/gradefor student for the year as post data.RELIABILITY/VALIDITY: F&P has been extensively field tested with a gradient of text difficulty basis that has been developed over 20 years. Course grades are indicators or predictors of high school completion and student success in post-secondary academics (American Institute for Research, 2013). Each student will only be reported once, either assessment or grade. TARGET: The number of students improved is ambitious for student starting point, student and school conditions, and data availability. City Year is currently working with the Center for Assessment to further understand the impact of learning loss due to COVID-19 and how this will impact our expectations and definitions of improvement.
ED5A: Number of students with improved academic performance
ED5A Outcome:
Report Card Grade, Standardized TestMeasured By:
322Target: Students
-
DRAFT
Page 24
For Official Use Only
Problem Statement:
Selected Interventions:
MentoringOther Classroom Support
Academic Services in MathPerformance Measure:
Education K-12 SuccessFocus Area:
Objective:No of Members:
82.00 130No of MSY's:
Describe Interventions:
DESIGN: City Year New York members are embedded in 3rd to 10th grade math classrooms where they serve as Student Success Coaches and provide focused academic mentoring in math and whole class support. In classrooms, members support the whole class in following instructional routines and provide small-group focused support using NYCDOE math protocols. Mentoring includes improving durable math skills and supports development of good study habits, such as note taking, collaboration, task prioritization, and homework completion. Member activities improve student on-task behavior and limit peer distraction while addressing student motivation and academic engagement, such as in-class assignment completion and participation. Members also model good academic behaviors and reinforce the concepts of a collaborative classroom climate. Outside of the classroom, members develop a culture of math through math-based events and enrichment programming, e.g. Pi Day, etc.DOSAGE: All students in the classroom are substantially engaged through full-time placement of members in the classroom for the entire academic year. Classes are 45-60 minutes per class five days a week. Focused math mentoring occurs in 20 to 60-minute sessions 2x per week. Each student included under ED1A is expected to receive at least 40 minutes of individualized support per week. Members may also provide math support to students before or after school. Once enrolled in our program, students are expected to receive
In 2021-22, City Year New York anticipates partnering with 20 elementary, middle and K-8/6-12 schools (of which, 5 are still being determined) within the neighborhoods of Queens, Harlem, Brooklyn and the Bronx, in order to improve student outcomes that lead to improved graduation rates. In 2018-19, City Year’s proposed partner schools enrolled nearly 7,000 students, the majority of whom represent minority (avg. 95%) and economically disadvantaged (avg. 87.4%) backgrounds (2018-19 School Performance Dashboard, NYC DOE). Annual state testing shows that low-income and minority students perform lower on state standardized testing for math than their more affluent and white peers. As a result, these students are at risk for not graduating high school on time and are in need of additional academic support. On the 2018-19 New York State Testing Program (NYSTP), 64% (3,166/4,963) of students tested in 3rd-8th grades at City Year partner schools scored below proficiency (Partially/Not Proficient) on the math portion of the exam. In comparison, 53%(505,607/948,606) of 3rd-8th grade students statewide, and of these, 54% (208,480/383,235) of students citywide tested below proficiency in math this same year (2018-19 School Performance Dashboard, NYC DOE).The city’s overall graduation rate has steadily increased since 2005 to a high of 77.3% in 2019, but is still significantly lower than the state’s graduation rate of 83.4% (2019 NY State Graduation Rate Data). The Early Warning Indicators predict that students who are below proficiency or failing math are at high risk to drop out (Balfanz & Neild, 2006). Further, early intervention in math before the 9th grade is key to getting a student who is demonstrating one or more of the four key performance indicator areas (English, math, attendance, and behavior) back on track for graduation (Corrin, et al, 2016).
-
DRAFT
Page 25
For Official Use Only
Problem Statement:
Selected Interventions:MentoringOther Classroom Support
Academic Services in MathPerformance Measure:
Described Instrument: TITLE: CYSCHOOLHOUSE. cyschoolhouse is a secure cloud-based platform thatprovides an intervention management and tracking system to record student-level data.WHAT IS MEASURED: Student profile data includes name/identifying number, school, grade level, member assignment, which targeted interventions students receive, and participation. The Gradebook module of cyschoolhouse tracks individual student grades and an Assessment module tracks attendance data, other school-based data, and assessments.HOW ADMINISTERED: cyschoolhouse is managed by City Year Headquarters. City Year New York staff enter student profile and school data with records provided by schools/district. Intervention data is entered by members using computer, tablet, or mobile phone application.RELIABILITY/VALIDITY: Only one record per student is created to ensure non-duplication, further City Year Headquarters cleans data to make sure that recordsare valid and not duplicated. Data is analyzed by both City Year New York and Headquarters analytics staff. cyschoolhouse is FERPA compliant and was designed by Acumen Solutions for City Year in 2012 and vetted by a consortium ofDC Charter schools. TARGET: Approximately 107 members will be assigned to math classrooms, wherethey support all students in the classroom (25-30 students per member) and are assigned up to 8 Focus List students for additional intensive math support above standard strategies, e.g. outside of class. Targets are based on the subset of students assigned to members in math classrooms and reflect variations in school enrollment size, class schedule, and member capacity. Team Leaders do not carry assigned Focus List students but engage all Focus List students in the
ED1A: Number of individuals served
ED1A Output:
Tracking SystemMeasured By:
890Target: Individuals
math services for the duration of the school year unless they withdraw from school or are assigned to another classroom. Combined class and focus time, students receive a minimum of 4.5 hours math services in math perweek.NYCDOE PROTOCOLS: Protocols follow the five strands of mathematical proficiency: Conceptual Understanding, Procedural Fluency, Strategic Competence, Adaptive Reasoning, and Productive Disposition, derived from the National Research Council and the IES Practice Guide.
-
DRAFT
Page 26
For Official Use Only
Problem Statement:
Selected Interventions:MentoringOther Classroom Support
Academic Services in MathPerformance Measure:
Described Instrument:
Described Instrument:
school.
TITLES: Outcome instruments are selected in partnership between City Year NewYork, the New York City Department of Education, and individual schools. Instruments: HMH Math Inventory (MI) or math course grades.WHAT IS MEASURED: All instruments measure student level progress in Math. Math Inventory measures skills and knowledge in math strands as applicable to the grade level such as, counting and cardinality, fractions, operations, ratios, proportional relationships, and algebraic thinking. Grades measure content mastery, homework completion, and participation. Improvement is defined as meeting or exceeding typical growth for the grade level and starting point of the student as established by the assessment vendor (MI) or improving course gradeor course grade point average (depending on alpha or point grading system).HOW ADMINISTERED: MI is a computer adaptive test administered by the schoolthree times a year. Grades are issued by teachers at least four times a year. The first assessment or marking period grade or, when available, prior year end of yearresults, will serve as baseline/pre data and the last assessment or grade for the student for the year will serve as post data.RELIABILITY/VALIDITY: MI is a product of Houghton Mifflin Harcourt, is nationally normed, and uses the research-based MetaMetrics quantile scoring system. Course grades are indicators or predictors of high school completion and student success in post-secondary academics (American Institute for Research,2013). Each student will only be reported once, either assessment or grade. TARGET: The number of students improved is ambitious for student starting point, student and school conditions, and data availability. City Year is currently working with the Center for Assessment to further understand the impact of learning loss due to COVID-19 and how this will impact our expectations and definitions of improvement.
ED5A: Number of students with improved academic performance
ED5A Outcome:
Report Card Grade, Standardized TestMeasured By:
312Target: Students
-
DRAFT
Page 27
For Official Use Only
AmeriCorps Funding Priorities
Grant Characteristics
*Check any priority area(s) that apply to the proposed program. In order to receive priority consideration, applicants must demonstrate that the priority area is a significant part of the program focus, high quality program design, and outcomes.
*Check any characteristics that are a significant part of the proposed program.
Program Information
COVID-19 Recovery, Evidence-Based Intervention
Geographic Focus-Urban
-
DRAFT
Page 28
For Official Use Only
Other Revenue Funds
Number of episodic volunteers generated by AmeriCorps members
Ed Priority: Enter row number (1-13) of intervention in Education Evidence Brief (enter 0 for N/A)
EO Priority: Enter row number (1-4) of intervention in Econ Opp Evidence Brief (enter 0 for N/A)
HF Priority: Enter row number (1-5) of intervention in Healthy F. Evidence Brief (enter 0 for N/A)
4156500
200
0
0
0
Demographics
-
DRAFT
Page 29
For Official Use Only
Required Documents
Evaluation
Labor Union Concurrence
Other Documents
Evidence Report - PSA Study
Evidence Report - Everyone Graduates
Document Name
Sent
Not Applicable
Not Applicable
Sent
Sent
Status
-
Page 30
Logic Model
The community problem that the program activities (interventions) are designed to address.
STUDENT SUCCESS COACHES: Academic performance in English and math are leading indicators of student drop out risk and students who are academically behind in middle
Resources that are necessary to deliverthe program activities (interventions), including the number of locations/sites and number/type of AmeriCorps members.
255 full-time AmeriCorps members, placed inschools (34-36 weeks), Mon-Fri.; 2 days/month are for training.
45 City Year staff positions
20 public schools receiving teams of
The core activities that define the intervention or program model thatmembers will implement or deliver, including duration, dosage and target population.
All Focus List students receive support services for the full school year:
A) ELA/Math support: academic mentoring 2x/week,20-60 min/session; extra support in daily class periods;
Direct products from program activities.
2,550 Focus List students in 3 to 10th grade, of which:
1,072 receive English Language Arts focused support and mentoring (ELA ED1A)
890 receive Math
Changes in knowledge, skills, attitudes and opinions. These outcomes, if applicable to the program design, willalmost always be measurable during the grant year.
2,550 Focus List students are routinely monitoredfor emerging trendsand improvements through our Student Progress Monitoring protocols, of which:
322 students will improve knowledge
Changes in behavior or action. Depending on program design, these outcomes may or may not be measurable during the grant year.
Students will improve behaviors that demonstrate commitment to learning and academic engagement
Students will improve behaviors in attendance
Students improve
Changes in condition or status in life. Depending on program design, these outcomes may or may not be measurable during the grant year. Some programs, such as environmental or capacity-building programs, may measure changes in condition over a period as short as one year.
80% of students who receive targeted services through middle school will not require targeted services in high school
Increased rates of on-time high school graduation
Problem Inputs Activities Outputs Short-Term Outcomes
Mid-Term Outcomes
Long-Term Outcomes
-
Page 31
For Official Use Only
Logic Model
school are less likely to succeed in high school. (Balfanz, Herzon, & Mac Iver, 2007; Kieffer & Marinell, 2012; Holcomb-McCoy, 2007; Levin et al, 2012). Students in City Year schools are performing below proficiency on stateand district readingand math assessments, and may demonstrate poor attendance or in-school behaviors. Students need supports to help improve individual academic skills, attendance, and social-emotional development.
members plus City Year staff supervisor (Impact Manager)
Member Skills and Professional Development /Observation and Coaching Program
Research-based Literacy and Math Protocols
Student Progress Monitoring System
Clover Youth Development Framework
Check In Check Out (CICO) coaching methodology
extended learning time and homeworkhelp.
B) Student Progress Monitoring; conducted bi-weekly with teachers, 1x/monthteam review.
C) Mentoring and goal-setting programs to improve behavior and social emotional development (daily).
D) Attendance monitoring and support programs (CICO; 20 min/week).
Target Population: At risk students in grades 3 to 10.
focused support and mentoring (Math ED1A)
597 receive attendance mentoring
2041 receive social emotional mentoring
and skills in Englishbased on assessments or course grades (ELAED5A)
312 students will improve knowledge and skills in Math based on assessments or course grades (Math ED5A)
149 students will improve their presence in school based on school attendance records
behaviors in self- management, interpersonal behavior and growth mindset
Increased rates of college enrollment
Problem Inputs Activities Outputs Short-Term Outcomes
Mid-Term Outcomes
Long-Term Outcomes
-
Page 32
For Official Use Only
Logic Model
WHOLE SCHOOL: Developmental research shows that meaningful relationships and knowledge-building foster positive feelings of connection and belonging for students (Blum & Libbey, 2004) and that supportive school climates further increase connectedness to school (Kidron & Osher, 2010). However, research also shows that under-resourced schools have a reduced capacity tocreate a climate that is conducive for stude