Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of...
Transcript of Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of...
![Page 1: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/1.jpg)
Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human
Services and Entire School Populations to Investigate What Works and for Whom
45-Minute Paper Presented at University of South Florida’s 31st
Annual Research & Policy Conference: Child, Adolescent, and Young Adult Behavioral Health, 4-7 March, 2018, Tampa, Fl.
Mansoor A. F. Kazi, PhD, President, Realist Evaluation Inc., Director Program Evaluation Center, Fredonia SUNY;Rachel M Ludwig, LCSW, Chautauqua Tapestry Director;Susan Hoerter, DO, Rockland County SOC Expansion Grant PI;Janet Sliva, Rockland County SOC Expansion Grant Manager
Marie McLaughlin, Director, Manchester Youth Justice Service, England
![Page 2: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/2.jpg)
Acknowledgements
• Dr. Bret Apthorpe, Superintendent; Jessie Joy, Director of Curriculum, Instruction & Assessment; Tina Sandstrom, Director of Schools, Jamestown Public Schools; John Panebianco, Director of Pupil Services
• Dr. Gary M. Blau, Chief, Child, Adolescent and Family Branch, Center for Mental Health Services, Substance Abuse and Mental Health Services Administration
• Patricia Brinkman, Commissioner; ChautauquaCounty Department of Mental Hygiene, Mayville
• Manchester City Council—Marie McLaughlin,Director Youth Justice Services
![Page 3: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/3.jpg)
Acknowledgements
• Tom Zugibe, District Attorney
• Michael Leitzes, Mental Health Commissioner
• Mary Jean Marsico, Rockland BOCES
• Dr. Susan Hoerter, Rockland SOC Expansion PI
• Janet Sliva, Rockland SOC Expansion Grant Manager
• Barbara Gavin, Director, Child Welfare Services
• Carol Korngold, Safe Youth Center Executive Director
• Kathy Farber, Safe Youth Center Consulting Director
• Kathy Tower-Bernstein, Director of Probation
![Page 4: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/4.jpg)
Tapestry of Chautauqua
County, New York
Mansoor A. F. Kazi, President, Realist Evaluation Inc.; Affiliate Research Fellow, University at Buffalo; Director, Program Evaluation Center, Fredonia State University of New York; Adjunct Fordham University
[email protected] [email protected]• American Evaluation Association (eval.org) Co-Chair
Human Services Evaluation Topical Interest Group & Social Work Topical Interest Group.
Based on Kazi, M. A. F.
(2003) ‘Realist Evaluation
in Practice’, London: Sage
Realist Evaluation Partnerships
SAMHSA’s
Gold Award for
Outstanding
Local
Evaluation
2010 Training
InstituteSTAKES/THL
MANCHESTER CITY
COUNCIL, England
ROCKLAND COUNTY NY
![Page 5: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/5.jpg)
New York State System of Care SAMHSA Funding
Funded by SAMHSA—Substance Abuse and Mental Health Services Administration, Center for Mental Health Services:
• ‘Tapestry of Chautauqua’ Systems of Care Grant (with Chautauqua County Department of Mental Hygiene, $9,000,000 2008-2015; $977,317 for evaluation, Funded by SAMHSA; Recipient of SAMHSA’s Gold Award for Outstanding Local Evaluation, July 2010).
• ‘Tapestry of Chautauqua’ Systems of Care Expansion Grant (with Chautauqua County Department of Mental Hygiene, $4,000,000 2015-2019)
• Rockland County (NY) Systems of Care Expansion Grant, with Rockland County, $4,000,000 2016-2020; $800,000 for evaluation, Funded by Substance Abuse and Mental Health Services Administration (SAMHSA).
![Page 6: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/6.jpg)
![Page 7: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/7.jpg)
Take the
Gary
blauchallenge
![Page 8: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/8.jpg)
T= (V + B + A) x (CQI)2EBP’s & Clinical
Excellence
Cultural & Linguistic
Competence
Continuous Quality
Improvement
Family Driven
Youth Guided
Transforming Children’s
Mental Health in America!
System of Care: Focus on Values
![Page 9: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/9.jpg)
Children’s Mental Health Initiative (CMHI):
Systems of Care
CMHI has impacted 22% of
the nations counties and 33
federally recognized Indian
Tribes
Since 2000, SAMHSA
administers an average of 61 system of care grants per year
CMHI sites have served over
100,000 children and youth
There are currently 88 CMHI funded
sites
![Page 10: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/10.jpg)
Systems Of Care Work!
Reduced behavioral and emotional problems
Increased behavioral and emotional skills
Reduced suicidal ideation and attempts
Reduced substance use problems
Improved functioning in school and in the community
Improved ability to build relationships
![Page 11: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/11.jpg)
View all of the KSOC-TV webisodes on the SAMHSA YouTube channel or by going to www.samhsa.gov/children
![Page 12: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/12.jpg)
The Goals of a System of Care
• To improve access and expand the array of services and supports for children and youth with a serious emotional disturbance and their families.
• To promote the full potential of every child and youth by addressing their physical, emotional, intellectual, cultural and social needs in family, school & community
![Page 13: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/13.jpg)
Who Comes to School?
• 21% of children ages 9-17 have a diagnosable mental health disorder (U.S. Surgeon General, 1999; NIMH, 20% currently)
• 20% of children with mental health disorders are identified and receive mental health services (U.S. Surgeon General, 2000; mentalhealth.gov—20% currently)
• 50% of students age 14+ who have a mental illness drop out of high school…the highest dropout rate of any disability group (U.S. Department of Education, 2001) (NIMH currently)
![Page 14: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/14.jpg)
Chautauqua Tapestry
• Why “Tapestry”?
• Quick history
• Goals
• Partners
![Page 15: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/15.jpg)
Chautauqua County, NY
![Page 16: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/16.jpg)
SAMHSA’s Theory of Change
![Page 17: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/17.jpg)
Expansion = System Change
System of care expansion is not a “project”
Goal is sustainable systemic changes in child-serving systems to improve services and outcomes
Occurs with or without federal grant
Grants are “venture capital” to achieve larger system change
Linked to other system reforms
![Page 18: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/18.jpg)
Keys to Successful Expansion
1. Commitment of high-
level policy and decision
makers
2. Realistic goals
3. High-priority goals
4. Specific, concrete
strategies
5. Strong leadership to
manage implementation
6. Linkages with larger
system reforms in
environment
7. Cross-agency
partnerships
8. Commitment across key
stakeholders
9. Staff and resources
allocated to
implementation work
![Page 19: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/19.jpg)
Evaluation as Sustainability Strategy
• Similar organizations collecting quality data, but uncoordinated and isolated
• Similar organizations collecting quality data for the same outcomes in a coordinated and collaborative network
• Similar organizations collecting quality data that align under similar outcomes for broad community impact
![Page 20: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/20.jpg)
Collaboration Defined:
A mutually beneficial relationship between two or more parties who work toward common goals by sharing responsibility,authority and accountabilityfor achieving results.
![Page 21: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/21.jpg)
Evaluation Examples
• Jamestown Public School
• North Rockland School District
• Manchester City Council, UK
![Page 22: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/22.jpg)
Critical Realism and Outcomes
• Sayer (1984): Realism links theoretical abstraction with empirical research
• “Essentially eclectic, the realist approach seeks to move away from what may be now rather sterile epistemological debates surrounding scientific versus pluralistic and constructivist paradigms, using all methodologies as appropriate within the realist framework. The approach facilitates the interrogation of empirical data with a view to identifying the nature of specific structural influences on outcomes” (p.45).
• Realist evaluation embraces outcome measurement and also investigates the factors that influence the outcomes
![Page 23: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/23.jpg)
Realist Evaluation
• Realist evaluation seeks to evaluate practice within the realities of society, based on Emergent and Symbiotic elements of natural law.
• All systems are emergent and constantly in a state of evolution, along with the reality that we are all symbiotically connected to nature and to each other.
• Practice takes place in an open system that consists of a constellation of inter-connected structures, mechanisms and contexts.
• Realism aims to address all the significant variables… through a realist effectiveness cycle which links the models of intervention with the circumstances in which practice takes place (Rom Harre’s models, analogous with reality)
• Explanation at any one time requires further investigation andfurther explanation (continuous testing & development of RomHarre’s models)
![Page 24: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/24.jpg)
Purposes and Objectives of evaluation
• Research methods drawn from both epidemiology and efficacy research traditions in a realist evaluation
• Partnership with human service agencies & schools to investigate what works and for whom. Real live data mined from human service agencies is used to investigate the effectiveness of the human service interventions.
• Emphasis is on data naturally drawn from practice.
• Binary logistic regression as part of epidemiologic evidence based on association, environmental equivalence, and population equivalence.
• How evaluators and agencies can make the best use of the available data to inform practice.
![Page 25: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/25.jpg)
Purposes and Objectives of evaluation
• Research indicates that the reform of the system of care and the use of initiatives such as wraparound are effective in improving mental health and functioning in school (Kutash, 2006; Reback, 2010; Goldenson, 2011)
• However, most studies have focused on at risk groups rather than the total school populations.
• The purpose of this study is to undertake a 100% evaluation of all school-based services, utilizing data on the entire school populations, in a longitudinal study.
![Page 26: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/26.jpg)
Evaluation Strategy With Each Agency
• Evaluation resource and service for you and for each participating agency
• How to access and to use your own MIS data
• How to analyze this data repeatedly to inform practice
• Carried out with you and for you only—you decide who to share with and how to use the findings
![Page 27: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/27.jpg)
Realist Evaluation: What Interventions work & in what circumstances
• A combination of efficacy research & epidemiology traditions
• Management Information System (MIS) Data routinely collected but typically not used for evaluation in agencies
• Investigate interrelationships between outcomes, client demographics, client circumstances, & services provided (interventions)
• Methods such as binary logistic regression can predict the likelihood of effectiveness of an intervention in given circumstances
• Use findings at regular intervals to better target and develop services
![Page 28: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/28.jpg)
Education Partners in SOC
• Education plays a critical role in the development of children.
• Positive learning experiences help prevent emotional and behavioral problems.
.
![Page 29: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/29.jpg)
School Examples of Realist Evaluation
• 17 NYS school districts have participated
• 100% data from school + participated agencies
• What works and for whom in achieving school,
agency and system of care outcomes
• Examples of school data
Demographics Intervention Outcome
• Ethnicity• Gender • Lunch status, • Special education
• School based interventions
• Pre-school programs
• Mental health • DSS
• GPA/ State tests• Behavior incidents• Attendance• Drop out rates
Graduation rates
![Page 30: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/30.jpg)
Data analysis and utilization
• Single system & one group pre/posttest
• Comparison of outcomes between baseline andsubsequent periods and those receiving and notreceiving interventions
• Investigation of patterns between outcomes,demographics and interventions
• Binary logistic regression
• Data analysis in partnership with schools andagencies
• Utilization of findings to develop and improveservices for children and families
![Page 31: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/31.jpg)
Outcomes: Jamestown Public Schools
• Strengths and Difficulties Questionnaire (SDQ) April 2015 and April 2016
• Recorded Incidents of Behaviour 2014/15 and 2015/16
• North West Evaluation Association (NWEA) Measures of Academic Progress (MAP) Fall 2015 , Spring 2016 & Fall 2016 (grade levels K to 10)
![Page 32: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/32.jpg)
Universal Screening Tool: SDQ
• Strengths and Difficulties Questionnaire (SDQ)
• Brief measure of pro-social behaviour and psychopathology, 3-17 yr olds
• Goodman (2001)—reliability .73
• Five factors: emotional symptoms, conduct problems, hyperactivity, peer relationships and pro-social
• Grade levels K to 4 in April 2012, 2013, 2014, 2015 & 2016
![Page 33: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/33.jpg)
SDQ 2012 Normal Scores/Ethnicity
![Page 34: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/34.jpg)
SDQ 2012 Predictors for
abnormal/borderline total scores
![Page 35: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/35.jpg)
Predictors for Change from 2012 to 2013
Total SDQ
![Page 36: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/36.jpg)
Hyperactivity Level 2015 SDQ
![Page 37: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/37.jpg)
Hyperactivity Level 2016 SDQ
![Page 38: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/38.jpg)
Hyperactivity Level Change from 2015 to
2016 SDQ
![Page 39: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/39.jpg)
Hyperactivity Level Change from 2015 to
2016 SDQ: Predictors for improvement
![Page 40: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/40.jpg)
Hyperactivity Level Change from 2015 to
2016 SDQ: Striders Advocacy
![Page 41: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/41.jpg)
Hyperactivity Level Change from 2015 to 2016
SDQ: Jamestown Behavioral Health Clinic
![Page 42: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/42.jpg)
Tapestry Partners Again Predictors for
Improved Behavior in Schools• Recorded incidents of behavior Compared between
2014-15 and 2015-16 school years
• Overall mean was 0.96 in 2014/15 & 1.58 in 2015/16 recorded incidents per student ( n = 5134) in Jamestown Public Schools in both years
• Once again Tapestry partners’ services were predictors for improved behavior, this time several more!
![Page 43: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/43.jpg)
![Page 44: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/44.jpg)
Safari Improved Behavior
![Page 45: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/45.jpg)
Improving Measurement of Academic Achievement
• Northwest Evaluation Association (NWEA)
• Measures of Academic Progress Normative Datafor Reading and Mathematics
• More reliable than GPA
• Used three times a year up to grade level 10
• Beginning from Baseline Year 2015/16
• First two times used Rasch Unit Scale (RIT) butfor the third assessment will use expectedgrowth
• Of those proficient at level 3 or 4, 51.8%demonstrated growth that was less thanexpected in the first two reading assessments
![Page 46: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/46.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 47: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/47.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 48: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/48.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 49: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/49.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 50: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/50.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 51: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/51.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 52: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/52.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 53: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/53.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 54: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/54.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 55: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/55.jpg)
NWEA Reading Fall 2015 & Spring 2016
![Page 56: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/56.jpg)
NWEA Reading Fall 2015 & Spring 2016
Predictors for Improving RIT Score at Median 9
or Higher
![Page 57: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/57.jpg)
Plan for Chautauqua Tapestry 2.0(2015-2019)
– Chautauqua County the only one to receive SAMHSA’s Gold Award for Outstanding Local Evaluation.
– Tapestry 2.0 will deepen its evaluation with school districts and human service agencies in the county.
– Chautauqua County is a world-leader in realist evaluation, and to date 28 agencies (including 7 school districts) have collaborated
![Page 58: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/58.jpg)
![Page 59: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/59.jpg)
![Page 60: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/60.jpg)
Outcomes: North Rockland Schools
• Recorded Incidents of Behaviour- 2016/17 (grade levels K-12th)
• Istation Universal Screener- Fall 2016 & Spring 2017 (grade levels 3rd to 6th)
• Developmental Reading Assessment 2 (DRA2)- Fall 2016 & Spring 2017 (grade levels 1st- 4th)
• North West Evaluation Association (NWEA) Measures of Academic Progress (MAP) Fall 2015 , Spring 2016 & Fall (grade levels K to 10) from 2017/18 School Year
![Page 61: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/61.jpg)
North Rockland Schools 2017/18
![Page 62: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/62.jpg)
Istation Universal Screener- Fall 2016 & Spring 2017 (grade levels 3rd to 6th)
![Page 63: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/63.jpg)
Developmental Reading Assessment 2 (DRA2)- Fall 2016 & Spring 2017 (grade levels 1st- 4th)
![Page 64: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/64.jpg)
Recorded Incidents of Behavior 2016/17 (8.9%)
to First Period 2017/18 (4%; N = 8026)
![Page 65: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/65.jpg)
Plan for Transforming Rockland County(2016-2020)
– National Evaluation Requirements
– Local Evaluation Strategies
– Continuous realist evaluation of services utilizing entire Management Information Systems
– Department of Social Services
– County Mental Health Services
– Probation Service
– North Rockland and Nyack School Districts
– Continuous quality improvement
![Page 66: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/66.jpg)
Plan for Rockland County(2016-2020)
– Rockland County Evaluation will expand from the largest North Rockland Schools to include all 8 school districts (n = 41,000)
– One of 37 in NY, Board of Cooperative Educational Services (BOCES) plays central role with Partnership for Safe Youth
– 100% data from County Mental Health, Department of Social Services, Probation Service and other human service agencies, and merged with whole school big data
![Page 67: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/67.jpg)
Example: Manchester Youth Justice Service in England
• This interim report is based on electronic data
comprising of a 100% sample of 2,526 youth in
the 4-year period January 2011 to December
2014
• 76.3% were male, and 62.2% of the youth were
white
• The average age at date of arrest for all youth
was 15 years, ranging from 10 to 18 years.
![Page 68: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/68.jpg)
Manchester Youth Justice Service in England
• Offence database recorded every episode of an
offence reported on the youth,
• Work out those that had offended after the first
recorded offence, and those that had more than
one, in the 4-year period from January 2011 to
December 2014.
• It was found that 66.1% had not offended again
after the first recorded offence (as compared
with 45.6% in the previous 10-year analysis
reported at AEA 2012).
![Page 69: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/69.jpg)
Binary Logistic Regression Model
![Page 70: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/70.jpg)
Predictors for returning
• Age at first order. Those older by every year were 1.5
times more likely not to return
• Gender. Females were more likely not to return than
males.
• Those who had completed the order were 7 times more
likely not to return.
• Youth living with parents or relatives were 2.6 times more
likely not to return.
• Those with higher baseline ASSET scores were more
likely to return
• Those on youth caution more likely not to come back
![Page 71: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/71.jpg)
Manchester Youth Justice Service:Linking Data with Practice
• Using our data differently
• Clearer picture of our cohort
• Understanding what is working andwhat is not working
• Helping to target resources moreeffectively
• Identifying predictors of re-offending
![Page 72: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/72.jpg)
Manchester Youth Justice Service:Looking Forward
• Improving the way we record data
• Working with partners to improve useof data
• Merging this data with a wider pool ofdata within the City of Manchestere.g. schools, police, health, benefitsagency
• Targeting resources more effectivelyto get better outcomes
![Page 73: Yes We Can: Continuous Realist Evaluation of …...Yes We Can: Continuous Realist Evaluation of System of Care Expansion by Combining Big Data from All Human Services and Entire School](https://reader034.fdocuments.us/reader034/viewer/2022042709/5f440f4d908723753400fdc1/html5/thumbnails/73.jpg)
Conclusion
• Realist evaluation strategies help to utilise theavailable whole school data and to conduct deeperanalysis
• Explanations for differences between those thatimprove on an outcome and those that do not
• Identifies patterns in the data where multiplefactors are influencing the outcome, and selectsthe main factor or factors responsible for theoutcome, with a prediction of the odds of achievinga given outcome in particular circumstances
• Repeatedly investigate changes in outcomes,intervention & the contexts