The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson,...
-
Upload
gordon-joseph -
Category
Documents
-
view
214 -
download
0
Transcript of The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson,...
![Page 1: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/1.jpg)
The Community Schools Evaluation Toolkit:
Moving the Research Agenda Forward
Reuben Jacobson, University of Maryland
Shital C. Shah, Coalition for Community Schools
![Page 2: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/2.jpg)
Agenda
www.communityschools.org
2
Research summaryCommunity Schools Evaluation ToolkitDiscussionWork time: work on your evaluations and
network
![Page 3: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/3.jpg)
Research Summary
www.communityschools.org
3
Are community schools effective at improving outcomes for students, families, and the
community?• Collected 153 studies of community school
and community school-like initiatives– CS models: lead-agency, community agency/CBO,
university-assisted, and staff-initiated– Scale: boutique, local, state, national
• No studies in peer-reviewed journals
![Page 4: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/4.jpg)
Characterizing the Literature
www.communityschools.org
4
• Mix of internal/external evaluators• Great variation in study design– Process/implementation studies (e.g., number of
students served)– Outcome studies (e.g., improved achievement)
• Number of outcomes depends on the unique strategies of each community school – found at least 30 outcomes– Tied to theory of action
• Selected 22 quasi-experimental studies (comparison group, interrupted time series, controls)
![Page 5: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/5.jpg)
Achievement
www.communityschools.org
5
The most rigorous studies indicate that community schools are improving student achievement for some students.
• High-implementing CIS schools scored +6% than comparison non-CIS schools in percent proficient in grade 8 math (p<.01) and +5.1% in grade 8 reading (p<.05)
• Mixed results out of Chicago for school-level data– Student-level demonstrates that OST participation matters
• More students who participated in Children’s Aid Society (CAS) after-school programs demonstrated a steady increase from 2004 to 2007 in their math performance levels as measured by the state assessment compared to students who did not attend CAS activities (p<.05)
![Page 6: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/6.jpg)
Attendance
www.communityschools.org
6
The most rigorous studies indicate that community schools are improving attendance.
• CIS: High-implementing elementary schools had higher attendance levels than comparison group (+0.2%, p<.05). So did high-implementing high schools (+0.3%, p<.01)
• CAS: students who participated in CAS after-school programs for 3-4 years had better attendance than students with lower or no participation (p<.05)
• SF Beacons: participants who attended 30+ days of Beacon after-school had 3.9% less total days of unexcused absences than those who participated less than 30 days (p<.001)
![Page 7: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/7.jpg)
Graduation Rate
www.communityschools.org
7
While a focus of many community schools, there is little evidence of improving the graduation rate.
• CIS: high-implementing community schools had a significantly better graduation rate than comparison schools (+4.8 percent, p<.01)– Compared to other large-scale dropout programs, high-
implementing CIS schools had the highest effect size for graduation rate (ES=.31) and second-highest for dropout rate (ES=.36)
• CPS: schools that have been in CSI longer have better rates of ninth-grade students “on-track” to graduate than comparison group (p<.05)
![Page 8: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/8.jpg)
Behavior
www.communityschools.org
8
While a focus of many community schools, there is little evidence of improving student behavior.
• CPS schools had significantly less disciplinary incidents than their matched comparison group from 2002 to 2006
• CAS teachers reported that they saw greater improvement in students getting along with others for students with higher rates of participation in CAS than comparison students during the 2006-2007 school year (significance not reported)
![Page 9: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/9.jpg)
Other Outcomes
www.communityschools.org
9
Overall, few studies measure, evidence is weak, for the following outcomes: Parental engagement Relationships with adults Student engagement Impact on instruction
![Page 10: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/10.jpg)
Future Directions for Evaluating Community Schools
www.communityschools.org
10
• Utilize high-quality study designs that establish causality Why? Funding. Have to demonstrate effectiveness and grant-makers
are increasingly aware of study design (e.g., i3)• Evaluate the CS strategy, not just programs
Community schools are more than after-school programming Is the strategy or particular programming causing changes in
outcomes?• Use appropriate unit of analysis (school, student, or both)• Measure:
level of implementation: fidelity to design matters (e.g., CIS) longevity intensity of the intervention (after-school participation, receiving
health services, …• Evaluate outcomes for families and community; also student
health• Develop standard measures: e.g., how do you measure
student engagement?• Assess effectiveness of particular CS components to modify
and strengthen the design (e.g., do CS coordinators matter?)
![Page 11: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/11.jpg)
Community Schools Evaluation Toolkit
www.communityschools.org
11
Rationale for Results FrameworkJohn W. Gardner CenterCommunity School EvaluatorsCoalition for Community SchoolsJP Morgan Chase
![Page 12: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/12.jpg)
What is in the Toolkit?
www.communityschools.org
12
• 35 Page step-by-step manual • Coalition for Community Schools Logic
Model • Indicators for 5 short-term results• Indicators for 4 long-term results• Descriptions of school, city, county, and state
data that is available to sites• 45 surveys –for sites that are ready to collect
additional data• Examples from real sites
![Page 13: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/13.jpg)
Goals of the Toolkit
www.communityschools.org
13
• Introduce the Coalition for Community Schools (CCS) Results Based Logic Model
• Support sites to use existing data sources and collect additional data (if needed)
• Support sites to identify areas of success and areas in need of improvement
• Provide evaluation planning tools
![Page 14: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/14.jpg)
4 Parts to the Toolkit:
www.communityschools.org
14
1. Before you Start: Begin with the End in Mind
2. Get Ready: Prepare to Evaluate 3. Get Set: Designing the Evaluation4. Go!: The Evaluation Process
![Page 15: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/15.jpg)
www.communityschools.org
15
![Page 16: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/16.jpg)
Discussion
www.communityschools.org
16
Describe your experience evaluating CS.What challenges have you experienced?
Successes?What are your evaluation needs?
![Page 17: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/17.jpg)
Work Time
www.communityschools.org
17
Work on your evaluations and network
![Page 18: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/18.jpg)
How to Get the Toolkit
www.communityschools.org
18
• Coalition for Community Schools Web site: www.communityschools.org
• Questions? Comments?• [email protected]
![Page 19: The Community Schools Evaluation Toolkit: Moving the Research Agenda Forward Reuben Jacobson, University of Maryland Shital C. Shah, Coalition for Community.](https://reader030.fdocuments.us/reader030/viewer/2022032600/56649dbc5503460f94aadf74/html5/thumbnails/19.jpg)
Contact information
www.communityschools.org
19
Reuben Jacobson [email protected]
Shital C. Shah [email protected]