Csde final class ppt 1
description
Transcript of Csde final class ppt 1
1
An Analysis of Quality Review Alignment with Selected
Governance Models
Connecticut Quality Review TeamJamie Alter, Gabriella Barbosa, Ranjana Reddy, Ryan Thorpe, William David Williams
April 8, 2013
2
Agenda
• Overview of Connecticut Quality Review project
• Design choices made to align Connecticut Quality Review with experimentalism
• Challenges in designing an experimentalist Quality Review system
• Examination of an unresolved question: Should CSDE use Quality Review for school accountability?
3
Created first iteration of QR standards, tools, and process based on research and effective practices
The team supported the Connecticut State Department of Education by designing standards, tools, and a strategic process for piloting Quality Review.
Conducted focus groups with CSDE to solicit feedback
Strategically recruited districts to participate in pilot
Solicited feedback on QR from district leader
Designed Reviewer and School Guides
Supported training of reviewers and orientation for schools and district personnel
2. Develop and implement a rigorous
instructional program that meets the learning needs
of all students
• High quality schools evaluate and select appropriate strategies, plans, instructional programs, and curriculum for their circumstances to support college and career readiness for all students.
• Similarly, they organize themselves so that school leaders are helping to facilitate continuous improvement in educator effectiveness at meeting learning goals.
3. Create a school culture and climate that enables
all students to learn
• High quality schools promote a positive learning environment and engage families and the community in continuous improvement.
Rationale
• High quality schools have an overall vision focused on a pathway to college and career readiness for all students that has generated a thoughtful theory of action, comprehensive data gathering, and careful plans for implementing the theory of change.
1. Create and monitor a vision and theory of
action focused on student learning
4. Align management and operations to facilitate
achievement of student learning goals and steadily improve
• High quality schools continuously monitor and improve management practices to ensure the are effectively serving student learning as the school’s primary objective
During the QR process, reviewers will assess schools against four overarching standards.
4
Overview of Standard
Standard Indicator (To what extent do schools…)
Student Learning-Centered Culture
Continuously-Improving Instruction
1.1 Create a Shared Vision for Student Learning
1.2 Develop Theory of Action and Goals
1.3 Target Student Needs
1.4 Assess and Monitor Progress
2.1 Develop Rigorous Curriculum
2.2 Build Instructional Capacity
2.3 Develop Teacher Leadership
2.4 Promote Evidence-based Instruction
These standards are aligned to 17 indicators of school practice.
Standard Indicator (To what extent do schools…)
Student and Family Investment
Operational Alignment to
Goals
3.1 Establish Welcoming Environment
3.2 Engage Teachers and Staff
3.4 Communicate with and Engage Families
4.1 Align Resources and Policies to Goals
4.2 Monitor and Revise Operations
3.5 Build Community and Family Partnerships
4.3 Manage Talent
4.4 Develop and Pursue Innovative Solutions
3.3 Engage Students
These standards are aligned to 17 indicators of school practice.
The Quality Review process involves pre-review data collection, the 2-day onsite review, and post-review feedback.
Pre-review Review Post-review
Description:
Key steps:
Reviewers and schools prepare for the formal process of data collection by gathering documents, examining school data, self-evaluating, and setting logistical parameters.
During a two-day school visit, reviewers collect and synthesize evidence from a variety of sources to gain a full picture of school performance against the set of standards for effective schools.
Reviewers provide verbal and written feedback to schools based on evidence collected during the quality review and facilitate a structured debrief and reflection on results.
• Principal Orientation• School Self-Evaluation• Conversation with District
Staff• Planning Conversation Among
Reviewers• Planning Conversation with
School Leader
• Classroom Observations• Constituent Interviews• Observation of Teacher Teams• Reviewer Check-ins• Principal Debriefs
• Presentation of Findings• Goal Setting and Planning • Rating and Scoring• Written Report• Verification• Reconsideration, if necessary
The pilot will include two teams of reviewers conducting quality reviews in 4 East Hartford schools in April/May.
Reviewer Team 1
Reviewer Team 2
School 1
School 2
School 3
School 4
REVIEWER TEAMS• Include at least 1
state and 1 district reviewer
• Trained on the QR standards and process by CSDE vendor
• Have experience as educators and/or educational leaders
SCHOOLS
• Four elementary schools; identified by CSDE as Focus Schools
• Located in Alliance District
• District has 61% FRL, 47% proficient reading, 64% proficient math (Grade 3 CMT)
9
The QR pilot is designed to accomplish three key objectives.
To increase stakeholder support for Quality Review
• East Hartford will have an opportunity to test-drive quality review. • The CSDE will conduct outreach sessions with other Alliance Districts to
encourage more districts to engage in Quality Review in future years.
To troubleshoot and improve the
standards, tools, and process
• Reviewers, districts, and schools will provide feedback on the pilot tools, process, and reports.• This feedback will be used to revise the QR system and to inform
recommendations for roll-out.
To build capacity within the state
• State reviewers will be trained to conduct reviews, evaluate schools, write reports, provide verbal feedback to schools, and to ensure inter-rater reliability through internal training.• The CSDE will develop internal procedures for collecting and analyzing data
from quality reviews and managing the pre- and post- review elements of the process.• Personnel from East Hartford will gain experience as co-reviewers.
10
Agenda
• Overview of Connecticut Quality Review project
• Design choices made to align Connecticut Quality Review with experimentalism
• Challenges in designing an experimentalist Quality Review system
• Examination of an unresolved question: Should CSDE use Quality Review for school accountability?
11
CT QR was designed to inform an “experimentalist” problem-solving process at the school level, by providing schools with targeted feedback on areas of improvement
Enhance Strategic Feedback and Learning
Communicate and Link Strategic Objectives
and Measures
Plan, Set Targets and Align Strategic
Initiatives
Clarify and Translate Visions and Strategy
SCHOOL PROBLEM
SOLVING FOR IMPROVEMENT
CT QR will ensure that expertise flows from local learning to the center, and that the state, in turn, will accumulate and share local learning across schools.
State Knowledge Base of Effective
Practices
SCHOOL SCHOOL
SCHOOLSCHOOL
13
Other specific design features of CT QR were also intentionally linked to experimentalism.
How it Meets an Experimentalist ObjectiveFeature of CT QR
Strategic, Open-Ended Standards
The system is built around standards that exemplify effective schools, while being broad enough to generate fruitful conversation and learning among schools operating in different contexts.
Involvement of Other Stakeholders
Involvement of School Throughout Process
Processes such as self-evaluation, principal check-in, and goal setting meetings empower local school leaders to be a critical part of monitoring and adjusting their own performance.
Experimentalism allows for a diverse group of stakeholders to be included as part of an accountability system, including those usually left out of the conversation like parents and the community.
14
Minimal or no evidence
• Minimal process. School does not have in place any coherent system for regularly monitoring the effects of school operations and management systems on student learning outcomes.
Progressing toward standard
• Limited process. School has a plan for monitoring effectiveness and efficiency of operations and management systems for their effects on student learning outcomes, but inconsistently uses the information learned from this monitoring.
Meets standard
• Strategic process. School effectively and strategically plans and executes systems to regularly monitor and revise effectiveness and efficiency of operations and management systems to improve their impact on student learning outcomes.
Exceeds standard
• Strategic process and continuous improvement. School effectively and strategically plans and executes systems to regularly monitor and revise effectiveness and efficiency of operations and management systems to improve their impact on student learning outcomes.
Standard 4: Operational Alignment to Goals Indicator 4.2: Monitor and Revise Operations
15
Other specific design features of CT QR were also intentionally linked to experimentalism.
How it Meets an Experimentalist ObjectiveFeature of CT QR
Strategic, Open-Ended Standards
The system is built around standards that exemplify effective schools, while being broad enough to generate fruitful conversation and learning among schools operating in different contexts.
Involvement of Other Stakeholders
Involvement of School Throughout Process
Processes such as self-evaluation, principal check-in, and goal setting meetings empower local school leaders to be a critical part of monitoring and adjusting their own performance.
Experimentalism allows for a diverse group of stakeholders to be included as part of an accountability system, including those usually left out of the conversation like parents and the community.
16
Self-Review ensures the QR process is collaborative, and empowers stakeholders by helping them engage in a structured process of reflection on strengths and weaknesses.
Reflect:• How are we doing?• How do we know?• How did we get here?
Plan:• What are our goals?• How will we achieve them?• What support do we need?
Execute:• How do we put our plan into
action?• What more needs to happen
to make the plan a reality?
Monitor:• What progress are we
making toward our goals?• How well are we aligning
resources to achieving our goals?
Collaborate:• With whom do we need to
partner to support our goals?• Who needs to know about
our goals and progress?
17
Other specific design features of CT QR were also intentionally linked to experimentalism.
How it Meets an Experimentalist ObjectiveFeature of CT QR
Strategic, Open-Ended Standards
The system is built around standards that exemplify effective schools, while being broad enough to generate fruitful conversation and learning among schools operating in different contexts.
Involvement of Other Stakeholders
Involvement of School Throughout Process
Processes such as self-evaluation, principal check-in, and goal setting meetings empower local school leaders to be a critical part of monitoring and adjusting their own performance.
Experimentalism allows for a diverse group of stakeholders to be included as part of an accountability system, including those usually left out of the conversation like parents and students.
18
Constituent conversations allow reviewers to gain information about stakeholders’ perspectives and experiences with the school across multiple indicators.
Focus Group with Teachers
Focus Group with Parents
Focus Group with Students
19
Agenda
• Overview of Connecticut Quality Review project
• Design choices made to align Connecticut Quality Review with experimentalism
• Challenges in designing an experimentalist Quality Review system
• Examination of an unresolved question: Should CSDE use Quality Review for school accountability?
Tension #1: An experimentalist QR relies on autonomy at the school level, but autonomy is noticeably constrained.
RESOLUTION
By working with the state and districts to design and implement QR, we ultimately hope to build buy-in for necessary structural changes at the school level. Essentially, states and districts will see that schools have internal capacity to drive change and that they need additional autonomy to facilitate improvement through the QR process.
EXPERIMENTALIST PROPOSAL
Schools should have the discretion to develop context-specific strategies to close gaps identified by the QR system
BUREAUCRATIC REALITY
Schools have limited flexibility in allocating financial resources and guiding talent management constrain their ability to make changes advocated by QR
21
Tension #1 Resolution: Indicator 4.3 Managing Talent was adapted to acknowledge school-level constraints around talent management.
Develop and implement a strategy for managing human capital, including identifying, recruiting, mentoring, providing career-enhancing opportunities for, and retaining the most effective teachers and assigning teachers to areas where their skills best match student learning needs.
Develop and implement a strategy for marshaling resources to manage human capital, including identifying, recruiting, mentoring, providing career-enhancing opportunities for, and retaining the most effective teachers and assigning teachers to areas where their skills best match student learning needs, given each individual school’s contextual constraints.
Original Indicator Adapted Indicator
Tension #2: An experimentalist QR uses broad standards that are not prescriptive, but stakeholders want clear expectations.
RESOLUTION
The version of QR being piloted includes a rubric with broad, experimentalist standards and an evidence document with narrative-style vignettes as
examples of what "well-developed" looks like for each indicator.
EXPERIMENTALIST PROPOSAL
CPRL proposed a rubric with intentionally broad standards
to encourage a dialogue between reviewers and school
personnel and to resist the tool’s use as a compliance
mechanism.
BUREAUCRATIC REALITY
The CSDE and many school and district personnel believe
that providing specific standards with concrete
expectations for acceptable evidence ensures the process is fair and best communicates expectations to stakeholders.
Tension #2 Resolution: Revised QR includes evidence document with vignette-like narratives to provide examples of “well-developed.”
Indicator 1.1Create a widely shared vision of the progress students are expected to make while at the school to get and stay on track to graduate on time and become college and career-ready and foster a deeply embedded culture of high expectations committed to realizing that vision.
“The school effectively conveys high expectations to students and parents that result in a deep commitment to the school’s vision. Success is celebrated during “Honors Night” where teachers, parents and students are rewarded for exemplary work. The school provides parents with continuous information on how well their children are learning via report cards, progress reports and Engrade, an on-line reporting system. In an effort to better support students and effectively work with families, the school administers a student survey that aids the school to better understand the circumstances of different families and differentiate support accordingly. Guidance counselors work closely with families and actively assist in the completion of high school applications ensuring that students and families make good choices when selecting high schools. The emphasis on high standards and open exchange of information results in an atmosphere of care and support. Consequently, parents report appreciating the high expectations for attendance, good citizenship and academic effort within a highly inclusive and welcoming environment.”
Original version included only indicators Adapted version added vignette-style narratives
24
Agenda
• Overview of Connecticut Quality Review project
• Design choices made to align Connecticut Quality Review with experimentalism
• Challenges in designing an experimentalist Quality Review system
• Examination of an unresolved question: Should CSDE use Quality Review for school accountability?
The CSDE faces a decision about whether to use the results of Quality Review as part of its system for school accountability.
Factor in School Accountability
Purely Developmental
School ratings on the qualitative indicators would be integrated with the quantitative metrics currently used (state test scores and graduation rates) to classify schools and to determine which schools require intervention.
Schools would continue to be classified based on only quantitative metrics; Quality Review ratings will be provided only to school and district leaders, who will be encouraged to use the information to drive their own improvement.
Professionalism
Experimentalism
Performance Managemen
t
Experimentalism
Performance management suggests using QR for Accountability whereas professionalism would use QR purely developmentally.
QR as a Factor in School
Accountability
QR as a Purely Developmental Tool
Teachers and school leaders…. QR should be…
The CSDE’s decision about how to use Quality Review is closely related to its theory of action for school improvement.
If the CSDE believes…
Schools are likely to respond to QR for stakes by…
• “gaming” the quality review by prepping for the review and performing on the day of the site visit
Factor in student
accountability
• taking the standards more seriously and working harder to authentically improve
• be fair, valid, include both leading and lagging indicators, and provide schools with actionable information
• help the state identify schools that need the most support
• are professionals who, when provided with information and autonomy, are motivated to improve without sanctions
Purely Developmental
Systems of school accountability should….
• are motivated by fair and valid assessments of performance that are public and result in sanctions and interventions
Then…
Discussion teams
Team 1 Dara, Michael, Rohan
Team 2
Team 3
Seo Yun, Tom, Zahreen, Jason
Team 4
Jill, Sana, Brian, Jason
Team 5
Regina, Alia, Matthew
New Haven
Discussion Role Play• Take a point of view on whether Quality Review should be
part of Connecticut’s school accountability system and why• If so, how would this work?
– Publically reported– Factor in classifying schools– Factor in determining whether schools are closed or turned
around
• Split into groups of three or four and take the following roles:– Person who is youngest will represent a Teacher in the School– Person who is second youngest will represent the Leader of the
School– Person who is third youngest will represent the District
Superintendent– Person who is the oldest will represent the State Commissioner
• Each party should advocate for the use quality review that is best from the assigned perspective.
• Note that each entity likely has a different viewpoint regarding the purpose for conducting of quality review as well as which use of quality review best serves the stated purpose.
Task
Roles
30
Meets primary objectives
Create a common language around school quality
Improve accountability system
Likely Unlikely
Build capacity in districts and state
As you discuss whether QR should be a factor in school accountability, consider which use of QR is most likely to meet each of QR’s primary objectives.
Build support among educators and the public
ExplanationPurely
Dev.Part of Acct.
Source: Interviews with districts and vendors.
Encourage and enable authentic school improvement
Report out.
Teacher
Principal
Superintendent
State Commissioner
• To come
Factor in Accountability? Purely Developmental?
• To come
• To come • To come
• To come • To come
• To come • To come
Appendix
33
NCLB exposed poor performance, but is blunt and limited; Connecticut is developing a richer, more nuanced system.
NCLBUse status-based accountability and
identification of “schools in need of improvement” to motivate schools to
improve
Impose sanctions if schools do not improve
Use rich accountability and tiered classification to enable
and motivate schools to improve
Support low performing schools in the process of strategic
planning and intervention
Give targeted low performing schools the flexibility and
resources they need to improve
Provide all schools with tools and information needed to
facilitate improvement
Impose sanctions if schools do not improve
A richer system
34
Qualitative Review could be integrated into Connecticut’s new accountability system for all schools.
Quantitative Factors
SPI < 6464 < SPI < 88; did not meet
annual targets
64 < SPI < 88;met annual
targetsSPI > 88;
met state goals
Qualitative Factors
Well Developed Transition Progressing Excelling Excelling
Proficient Transition Transition Progressing Excelling
Developing Review Transition Transition Progressing
Under-developed Review Review Transition Transition
35
Qualitative Review could be integrated into Connecticut’s new accountability system as an additional component for schools on the cusp.
Excelling
Progressing
Transition
Review
Schools on the cusp of Excelling
Schools on the cusp of Progressing
Schools on the cusp of Transition
Schools receive QR for next three years
If score Proficient or Well-Developed, will be classified in the higher category even if quantitative data falls short by slight margin.
36
Qualitative Review could be integrated into Connecticut’s new accountability system as an additional requirement for exiting Focus and Turnaround status.
Focus Status
Turnaround Status
Lowest subgroup SPI or grad rates
Lowest SPI for “all students”
Meet subgroup SPI and grad rate targetsAchieve Proficient or Well-Developed QR Score
Meet SPI and grad rate targetsAchieve Proficient or Well-Developed QR Score