High Quality Evidence Impact Tool€¦ · Web viewHigh Quality Evidence of Impact Flowchart At a...

Formal Review – Submission High-Quality Evidence of Impact Advisory

The formal review process relies on an evidence base constructed by:

The Sponsoring Organization (SO) under review,

State level data Evidence collected during the onsite

portion of the review

Taken together, this collection of evidence helps the review team determine whether each program approval criterion is rated a commendation, met, or finding in the final report. During the offsite portion of the review, the burden of evidence rests solely with the SO to provide high-quality evidence on which decisions can be made. An SO organizes and submits evidence by responding to prompts in offsite domain worksheets. In all cases, evidence of impact is valued more highly than evidence that describes inputs or plans (see: Hierarchy of Evidence graphic).

The goal of this advisory is to support SOs’ understanding and self-reflection of what high-quality evidence of impact looks like in the offsite portion of the review. More specifically, this advisory will provide general guidance on how to plan for the incorporation of internal (SO-collected) and external (state-collected) data into the formal review. SOs provide a comprehensive body of evidence that may include quantitative evidence, data that can be counted or ‘quantified’, and qualitative evidence, data that describes or characterizes attributes. With increased access to quantitative data on Edwin Analytics and public profiles, SOs have many more data points to consider for their continuous improvement and accountability.

The flowchart below provides guidance to SOs to determine the sufficiency of the evidence provided in the offsite submission. As the review process places the most value on evidence of impact, the focus of the flowchart is impact evidence.

Note: This tool is not designed to be indicative of the overall rating of the offsite submission. Rather, it is intended to help SOs self-reflect on the strength and quality of the evidence of impact submitted for review. Evidence statements that are written in alignment with these components will maximize the amount of information that the review team is able consider in the decision-making process.

To further support SOs in incorporating evidence into the formal review, this advisory includes several appendices. Appendix A includes additional guidance about the flowchart. Appendix B provides a list of available state data and where to access them. Appendix C features additional guidance around the inclusion of quantitative data in the offsite submission. Appendix D provides a self-assessment by domain for SOs to evaluate the quality of their evidence of impact in the offsite submission. SOs can use the tool to refine and finalize the offsite submission

Educator Preparation Review – High-Quality Evidence of Impact Tool1

Hierarchy of Evidence

http://www.doe.mass.edu/edprep/toolkit/1617/Overview-DomainWsheets.pdf

http://www.doe.mass.edu/edprep/toolkit/1819/criterialist.pdf


Sufficient evidence of impact in the offsite portion of the review has the following components:

High Quality Evidence of Impact FlowchartAt a minimum, strong evidence of

impact is:Next, strong evidence of impact is: When possible, it should:

Outcomes-based Evidence focuses on the

results of the programmatic feature.

Aligned with Criterion Practices and evidence of

impact should be directly linked to the criterion being evaluated. It must address all aspects of the criterion.

If yo

ur e

vide

nce

is ou

tcom

es-b

ased

and

alig

ned

with

cr

iterio

n, m

ove

to th

e ne

xt c

ompo

nent

s.

Specific, and wherever possible, quantifiable Evidence of impact should be specific about

the result for each programmatic feature. It should be descriptive of the outcome and, when possible, numeric in measure.

Measured by multiple sources It is important to triangulate evidence from

multiple sources, whenever possible. SOs should consult the Review Evaluation Tools in order to see the multiple sources of evidence ESE uses to evaluate a single criterion.

Interpreted All evidence shared (whether it is state level

data or SO-specific data) should be explained in a way that articulates the key takeaway(s).

Capture multiple points in time This could include pre- and

post- data; formative and summative data; and/or longitudinal data from multiple points throughout a year or over multiple years.

Overall, strong evidence of impact is:Connected to plans and inputs

All evidence shared should be woven together in a cogent narrative that explains the connection between evidence of impact and inputs and clearly articulates any subsequent plans.

Educator Preparation Review – High-Quality Evidence of Impact Tool 2

Rig

http://www.doe.mass.edu/edprep/evaltool


Appendix A: Examples

The table below provides further guidance and examples of strong and weak evidence. Please note, these examples are not meant to be exhaustive.

Guidance Example(s) Non-Example(s)

Outcomes-based

What evidence tells you this feature/initiative/ input is successful? What are the results?

Student learning data Survey data Educator evaluation ratings Retention data Quotes from internal and external

stakeholders explaining the impact they experienced after the programmatic feature/initiative/input

o Statements of activitieso Syllabio Meeting minuteso Resumes/CVso Frequencies of meetings

Aligned with Criterion

Does the evidence speak specifically to what the criterion is measuring?

Uses language from the criterion Addresses all components of criterion

o Disconnected or broad statements

o Does not provide reviewer with information of the extent to which the SO is meeting criterion

Specific and, whenever possible, quantifiable

Is the evidence clearly indicative of the intended outcome/result?

“Survey data (see supplemental doc #1) indicates that 90% of candidates (n=67) said they were satisfied with the advising on licensure requirements they received in 2017.” (Scale: Satisfied, somewhat satisfied, somewhat dissatisfied, dissatisfied)

o “Many candidates have expressed satisfaction with advising.”

Measured by multiple sources

Do multiple sources of evidence corroborate or reinforce the claim? Are there discrepancies that need to be addressed?

Multiple surveys SO-collected data and state collected

data Data on the same outcome from

multiple stakeholder groups Quantitative and qualitative data points

related to the same outcome/result

o Evidence of an outcome/result is limited to data from a single data point, single source of evidence, or single stakeholder group.

Interpreted

Is the evidence written in a way that makes the conclusion clear to the reader? Would multiple different readers reach the same conclusion?

Shares the data point(s) and additional context to explain the takeaway

o Shares the data point(s) without additional context

Captures multiple points in time

Does the data articulate trends or patterns over

Pre-/post- tests that measure growth Graphical displays of progress Formative and summative data

o Single summative data pointo Multiple data points that do

not align with each other



Guidance Example(s) Non-Example(s)

time?Connected to inputs and plans

Are various data points clearly connected?

Makes explicit connection between data and practice

Identifies the problem and the impact of the solution

Tells the story of data driven decision making and resulting outcomes

o Lists of facts and/or data without connecting it to the context or prompt question



Appendix B: Data sources in Edwin

The graphic below indicates what data metrics are available in what Edwin report. For more information about the types of questions you can answer in each report, see the Edwin Overview page.


http://www.doe.mass.edu/edprep/edwin/overview.html


Appendix C: Sharing Quantitative Data in the Formal Review

ESE collects and reports on a number of output data measures; ESE and reviewers use several of these measures to inform decisions made in the formal review for program approval.1 SOs have the opportunity to share state-collected or SO-collected data along with additional context, analysis, and their own conclusions based on the data to demonstrate the extent to which they are meeting ESE expectations as outlined in the Program Approval Criteria.

Below are general guidelines to consider while incorporating quantitative data into the offsite portion of the formal review to ensure data included provides the review team with the information they need to incorporate the data in their decision making process:

Guidance Rationale ExampleShare data from the three most recent years

Although SOs may choose to include data from any year, by including the last three years of data at a minimum, reviewers get an indication of the program in its most current iteration.

“Candidates who completed between 2014 and 2016 agreed…”

Include the n-size This will help reviewers understand how representative this data is of the population of stakeholders.

“In 2014, we surveyed 8 completers and found that…”

If sharing survey data, include the entire survey scale for the item in the prompt response.2

It is important for reviewers to understand the full range of responses in SO-collected data.

“TCs are given Practicum Surveys exiting the program. 80% stated that they were very satisfied with their advising. (See supplement #1 Program Survey data)”

1 See Quick Reference Guide – Output Data (will be linked once posted on website)2 Note: This will not count against word limits.


http://www.doe.mass.edu/edprep/toolkit/1819/criterialist.pdf


Appendix D: High-Quality Evidence of Impact ChecklistHow to use this checklist: When constructing your offsite submission, use this checklist to help determine if you are submitting high-quality evidence of impact. It may be beneficial for someone who did not write the submission to complete the checklist for an objective perspective.

Organization: Is the organization set up to support and sustain effective preparation?

Evidence of Impact is: ORG1 ORG2 ORG3 ORG4 ORG5 ORG6 Notes/Examples

Outcomes-basedYes

PartialNo

Aligned with criterionYes

PartialNo

If “yes” for the two above, continue to next sessions. If “no,” stop and reconsider the evidence you are including.Specific and, wherever possible, quantifiable

YesPartial

No


YesPartial

No

InterpretedYes

PartialNo


YesPartial

No

Connected to plans and inputs

YesPartial

No


Formal Review – Submission High-Quality Evidence of Impact AdvisoryPartnerships: Is the organization meeting the needs of the PK-12 system?

Evidence of Impact is: PAR1 PAR2 PAR3 PAR4 PAR5 Notes/Examples

Outcomes-basedYes

PartialNo


PartialNo

Specific and, wherever possible, quantifiable

YesPartial

No


YesPartial

No

InterpretedYes

PartialNo


YesPartial

No


YesPartial

No


Formal Review – Submission High-Quality Evidence of Impact AdvisoryContinuous Improvement: Is the organization engaging in continuous improvement efforts that result in better prepared educators?

Evidence of Impact is: CI1 CI2 CI3 CI4 Notes/Examples

Outcomes-basedYes

PartialNo


PartialNo


YesPartial

No


YesPartial

No

InterpretedYes

PartialNo


YesPartial

No


YesPartial

No


Formal Review – Submission High-Quality Evidence of Impact AdvisoryCandidate: Is the candidate’s experience in the program contributing to effective preparation?

Evidence of Impact is: CAN1 CAN2 CAN3 CAN4 CAN5 Notes/Examples

Outcomes-basedYes

PartialNo


PartialNo


YesPartial

No


YesPartial

No

InterpretedYes

PartialNo


YesPartial

No


YesPartial

No


Formal Review – Submission High-Quality Evidence of Impact AdvisoryField-Based Experiences: Do candidates have the necessary experiences in the field to be ready for the licensure role?

Evidence of Impact is: FBE1 FBE2 FBE3 FBE4 FBE5 FBE6 Notes/Examples

Outcomes-basedYes

PartialNo


PartialNo


YesPartial

No


YesPartial

No

InterpretedYes

PartialNo


YesPartial

No


YesPartial

No


Formal Review – Submission High-Quality Evidence of Impact AdvisoryField-Based Experiences (cont’d)

Evidence of Impact is: FBE7 FBE8 FBE9 FBE10 FBE11 FBE12 Notes/Examples

Outcomes-basedYes

PartialNo


PartialNo


YesPartial

No


YesPartial

No

InterpretedYes

PartialNo


YesPartial

No


YesPartial

No


Formal Review – Submission High-Quality Evidence of Impact AdvisoryInstruction: Do candidates have the necessary knowledge and skills to be effective?

Evidence of Impact is: INS1 INS2 INS3 INSa INSb INSc Notes/Examples

Outcomes-basedYes

PartialNo


PartialNo


YesPartial

No


YesPartial

No

InterpretedYes

PartialNo


YesPartial

No


YesPartial

No


Formal Review – Submission High-Quality Evidence of Impact AdvisoryInstruction (cont’d)

Evidence of Impact is: INSd INSe INSf INSg Notes/Examples

Outcomes-basedYes

PartialNo


PartialNo


YesPartial

No


YesPartial

No

InterpretedYes

PartialNo


YesPartial

No


YesPartial

No


High Quality Evidence Impact Tool€¦ · Web viewHigh Quality Evidence of Impact Flowchart At a...

Documents

Transcript of High Quality Evidence Impact Tool€¦ · Web viewHigh Quality Evidence of Impact Flowchart At a...