The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle...
-
Upload
lorin-harmon -
Category
Documents
-
view
218 -
download
3
description
Transcript of The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle...
![Page 1: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/1.jpg)
The Process we Used and What we Learned….
Renee Bradley, Judy Shanley – OSEPHerb Baum, Danielle Schwarzmann – ICF Macro
2009 OSEP Project Director’s MeetingJuly 20th, 2009
05/03/23 1
![Page 2: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/2.jpg)
Thank you to those projects who participated
It’s never too early to think about Performance Measurement… Performance Measures: Under the Government Performance and
Results Act of 1993 (GPRA), the Department has established a set of performance measures, including long-term measures, that are designed to yield information on various aspects of the effectiveness and quality of the Technical Assistance and Dissemination to Improve Services and Results for Children with Disabilities program. These measures focus on the extent to which projects provide high quality products and services, the relevance of project products and services to educational and early intervention policy and practice, and the use of products and services to improve educational and early intervention policy and practice. The grantee will be required to provide information related to these measures. The grantee also will be required to report information on the project’s performance in annual reports to the Department (34 CFR 75.590).
Applications, work scope, reporting requirements
05/03/23 2
![Page 3: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/3.jpg)
Provides aggregate picture of our performance
Affects potentially, future program funding Supports profession of TA&D Enhances legitimacy of the field of TA&D
05/03/23 3
![Page 4: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/4.jpg)
Annual◦ Quality, usefulness, relevance, cost (efficiency)
Long-term◦ Implementation of evidence-based practices◦ Implementation of model demonstrations
05/03/23 4
![Page 5: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/5.jpg)
5
Annual MeasurePercentage of Special Education Technical Assistance
and Dissemination (TA&D) Products and services deemed to be of high quality by an independent (science) review panel (annual).
Substance:◦ Does the product or service reflect an evidence-based
approach or one grounded in current legislation or policy?
Communication:◦ Is the presentation of the product or service
description clear, well-formatted, organized?
05/03/23
![Page 6: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/6.jpg)
6
Annual MeasurePercentage of Special Education Technical Assistance and
Dissemination (TA&D) Products and services deemed to be of useful by an independent (science) review panel (annual).
Ease:◦ Is the product or service description easily understood,
providing guidance and direction? Replicability:
◦ Will the product or service eventually be used by the target group to achieve the benefit intended?
Sustainability:◦ Will the product or service eventually be used in more
than one setting and over time?
05/03/23
![Page 7: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/7.jpg)
7
Annual MeasurePercentage of Special Education Technical Assistance and
Dissemination (TA&D) Products and services deemed to be of relevant by an independent (science) review panel (annual).
Need:◦ Does the product or service solve an important
problem or address a critical issue? Pertinence:
◦ Does the product or service relate directly to the problem or issue facing the targeted group?
Reach:◦ Does the product or service apply to diverse
populations within the target group?
05/03/23
![Page 8: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/8.jpg)
8
OMB requests each Federal program to provide information on cost.◦ Cost is defined as the $ per unit of output.◦ Cost includes labor and other direct costs.◦ Cost includes only those expenses incurred by
Federal project.◦ For TA&D this is the cost of producing the ‘best’
practice/service.
05/03/23
![Page 9: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/9.jpg)
9
Implemented this year for TA&D program for the first time
Twenty-two TA&D projects and 10 State Deaf-blind projects randomly selected
Two Long-term Measures: ◦ Implementation of Evidence-Based Practices◦ Implementation of Model-Demonstration Projects
05/03/23
![Page 10: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/10.jpg)
10
Percentage of school districts and service agencies receiving Special-Education TA&D services regarding scientifically or evidenced-based practices for infants, toddlers, children and youth with disabilities that implement those practices (long-term).
05/03/23
![Page 11: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/11.jpg)
11
The percentage of TA&D projects responsible for developing models that identify, implement, and evaluate effective models (long-term).
05/03/23
![Page 12: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/12.jpg)
05/03/23 12
Dr. Herbert Baum,Danielle Schwarzmann
ICF Macro
![Page 13: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/13.jpg)
Selection◦ Random Selection
Using placemat◦ Additional sites suggested by OSEP to ensure
Evidence—based practice areas were represented Data Collection
◦ Initial e-mail◦ Follow-up/ Reminder e-mails◦ Phone calls when necessary
05/03/23 13
![Page 14: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/14.jpg)
1. E-mail requesting ‘best’ product/service (including costs), and list of services/products
2. Randomly selected a product/service as a ‘typical’ submission
3. Requested the program complete the ‘typical’ description
05/03/23 14
![Page 15: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/15.jpg)
Products/services are evaluated for 3 criteria1. Quality
Substance Communication
2. Relevance Need Pertinence Reach
3. Usefulness Ease Likelihood of Use Replicability
05/03/23 15
![Page 16: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/16.jpg)
OMB requests each federal program to provide information on cost.
Cost is defined as the $ per unit of output. For TA&D this is the cost of producing the
‘best’ practice/service.
05/03/23 16
![Page 17: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/17.jpg)
TA&D centers submitted forms indicating their practices/programs and where they were being implemented
ICF Macro chose a practice, and a place of implementation to complete a form
05/03/23 17
![Page 18: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/18.jpg)
Best Product/Service Description Typical Product/Service Description
Product Service Have not submitted
Response Rate Product Service Have not
submittedResponse
Rate
TA&D 17 5 0 100% 12 4 6 72%
Deaf/Blind 10 0 0 100% 7 2 1 90%
List of Products/Services Best Cost Measure 1.1
Submitted Have not Submitted
Response Rate Submitted Have not
SubmittedResponse
Rate Submitted Have not Submitted
Response Rate
TA&D 20 2 91% 17 5 77% 22 0 100%
Deaf/Blind 9 1 90% 10 0 100%
05/03/23 18
![Page 19: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/19.jpg)
Consider using electronic submissions - may not need to mail hard copies You save money on shipping Decreases response time Trees are saved
Submit accessible products - Diminish use of copywritten products as exemplars
Products – not links to websites must be submitted
Talk with your Project Officer and TA recipients about the Program Evaluation
05/03/23 19
![Page 20: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/20.jpg)
April◦ Final review of methodology by OSEP◦ Preparation of protocol materials
May◦ Select sample of projects◦ Send requests to projects
05/03/23 20
![Page 21: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/21.jpg)
June◦ Obtain description of ‘best” and “typical””
product and service◦ Obtain lists of products and services◦ Obtain products◦ Obtain cost data
July - Review by expert panels August – Generate Measures September – Report finings and
recommendations to OSEP
05/03/23 21
![Page 22: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/22.jpg)
Collect feedback from 2009 samples and use to improve process
Share results Discuss with grantees in kick-off meetings
and during the year Explore ways to integrate content from
annual continuation reports into the annual program evaluation
Learn from other Federal agencies Support and engage OSEP Project Officers
05/03/23 22
![Page 23: The Process we Used and What we Learned…. Renee Bradley, Judy Shanley – OSEP Herb Baum, Danielle Schwarzmann – ICF Macro 2009 OSEP Project Director’s Meeting.](https://reader035.fdocuments.us/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8ad/html5/thumbnails/23.jpg)
Comments, Questions, Concerns?◦ What could we do to improve our process?◦ What could we do to enhance definitions and
instructions?◦ What can we do to make life easier for you?
05/03/23 23