Stephanie Day, Deputy Budget Director Department of Administration, Budget Division.
Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.
-
Upload
barnard-poole -
Category
Documents
-
view
215 -
download
3
Transcript of Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.
![Page 1: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/1.jpg)
Using Data for Program Quality Improvement
Stephanie Lampron, Deputy Director
![Page 2: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/2.jpg)
2
Session Overview
The Title I, Part D Data Collection
Importance of Data Quality and Data Use
Actively Using Data for Program Improvement
![Page 3: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/3.jpg)
3
The Title I, Part D Data Collection
![Page 4: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/4.jpg)
4
What are Title I, Part D and NDTAC?
Title I, Part D (TIPD) of the Elementary and Secondary Education Act of 2001
– Subpart 1-State Agency
– Subpart 2-LEA
National Evaluation and Technical Assistance Center for the Education of Children and Youth who Are Neglected, Delinquent or At-Risk (NDTAC)
![Page 5: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/5.jpg)
5NDTAC's Mission Related to Data and Evaluation
Develop a uniform evaluation model for State Education Agency (SEA) Title I, Part D, programs
Provide technical assistance (TA) to States in order to increase their capacity for data collection and their ability to use that data to improve educational programming for N & D youth
![Page 6: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/6.jpg)
6Background: NDTAC’s Role in Reporting and Evaluation
Specific to Title I, Part D, Collections TA prior to collection
Webinars, guides, and tip sheets
TA during collection
Data reviews, direct calls, and summary reports for ED
Data analysis and dissemination
GPRA, Annual Report, and online Fast Facts
Related TA Data use and program evaluation
![Page 7: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/7.jpg)
7TIPD Basic Reporting and Evaluation Requirements
Where do requirements come from? Elementary and Secondary Education Act, amended in
2001 (No Child Left Behind)
– Purpose of Title I, Part D (Sec. 1401)
– Program evaluation for Title I, Part D (Sec. 1431-Subpart 3)
How does ED use the data? Government Performance and Results Act (GPRA) Federal budget requests for Title I, Part D Federal monitoring Provide to NDTAC for dissemination
![Page 8: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/8.jpg)
8Collection Categories for TIPD in the Consolidated State Performance Report (CSPR)
Types/number of students and programs funded
Demographics of students within programs
Academic and vocational outcomes
Pre-posttesting results in reading and math
![Page 9: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/9.jpg)
9
Title I, Part D in Pennsylvania
State Agency (S1) Local Agency (S2)
2008-09 2009-10 2010-11 2008-09 2009-10 2010-11
Number of Programs
US771 720 861 2,712 2,889 2,689
PA7 8 11 295 286 288
Number of Students Served
US125,456 109,146 106,747 373,071 367,121 354,591
PA 1,643 (1%)
1,189 (1%)
1,123 (1%)
24,863 (7%)
24,562 (7%)
26,510(7%)
![Page 10: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/10.jpg)
10Local Education Agency (S2) Academic Outcomes
* 2010-11 data are preliminary
![Page 11: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/11.jpg)
11Long-term Students Improvement in Reading (Subpart 2)
* 2010-11 data are preliminary
![Page 12: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/12.jpg)
12Long-term Students Improvement in Math (Subpart 2)
* 2010-11 data are preliminary
![Page 13: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/13.jpg)
13
Data Quality & Data Use
![Page 14: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/14.jpg)
14
14
Functions of Data
Help us identify whether goals are being met (accountability)
Tell our departments, delegates, and communities about the value of our programs and the return on their investments (marketing)
Help us replace hunches and hypotheses with facts concerning the changes that are needed (program management and improvement)
Help us identify root causes of problems and monitor success of changes implemented (program management and improvement)
![Page 15: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/15.jpg)
15
You need to TRUST your data as it informs:
Funding decisions
Technical assistance (TA) needs
Student/facility programming
15
Why Is Data Quality Important?
![Page 16: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/16.jpg)
16
16
What Is “high data quality”?
If data quality is high, the data can be used in the manner intended because they are:
Accurate Consistent Unbiased Understandable Transparent
![Page 17: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/17.jpg)
17
What data are the most useful?
Useful data are those that can be used to answer critical questions and are…
Longitudinal Actionable (current, user-friendly) Contextual (comparable, part of bigger picture) Interoperable (matched, linked, shared)
Source: Data Quality Campaign
Source: Data Quality Campaign
![Page 18: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/18.jpg)
18
18
Should you use data that has lower quality data?
YES!! You can use these data to…
Become familiar with the data and readily ID problems
Know when the data are ready to be used more broadly or how they can be used
Incentivize and motivate others
![Page 19: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/19.jpg)
19
Insure systems, practices, processes, and/or policies are in place− Understand the collection process
− Provide/request TA in advance− Develop relationships − Develop multilevel verification processes− Track problems over time− Use the data (even when problematic)− Link decisions (funding, hiring, etc.) to data evidence
Indicate needs to others 19
Data Quality Support Systems
![Page 20: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/20.jpg)
20
Using Data Actively
![Page 21: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/21.jpg)
21
Essential Steps Related to Data Use
1. Identify problem or goal to address
2. Explore & analyze existing data
3. Develop and implement change Set targets and goals
4. Develop processes to monitor and review data
![Page 22: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/22.jpg)
22
Step 1: Identify concerns or goals
Identify your level of interest State Facility / School Classroom
Define, issue, priorities or goals Upcoming decisions State or district goals or initiatives Information from needs assessments (or, conduct one)
Identify how data will be used & questions
Resource: NDTAC Program Administration Planning Guide-Tool 3 on Needs Assessments
![Page 23: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/23.jpg)
23
23
Program Components by Data Function
Program Accountability
Program Marketing/ Promotion
Program Improvement
Student demographics
Are the appropriate students being
served?
How are you addressing the
needs of diverse learners?
Which students need to be better
served?
Student achievement
Are students learning?
What are students learning? What gains have they
made?
How can we help improve student achievement?
Student academic outcomes
Are students continuing their
education?
What are students doing to continue their education?
How can we help improve student
academic outcomes?
![Page 24: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/24.jpg)
24
24
Focusing the Questions
Break the question into inputs and outcomes:
Inputs (what your program contributes):− Teacher education, experience, full-time/part-time
− Instructional curriculum− Hours of instruction per week
Outcomes (indicators of results): − Improved posttest scores
− Completed high school− Earned GED credentials
![Page 25: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/25.jpg)
25
25
Focusing/Refining the Question
Weak Question: Does my school have good teachers?
Good Question: Does student learning differ by teacher?
Better Question: Do students in classes taught by instructors who
have more teaching experience have higher test scores than those taught by new teachers?
![Page 26: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/26.jpg)
26
Step 2: Explore Existing Data
Locate the data you do have
Put it in a useful format−Trends, comparisons
What story is the data telling you?−What jumps out at you about the data?−Are the data telling you something that is timely and
actionable?−What questions arise? What is the data not telling
you that you wish you knew?**−What data could help answer those questions?
![Page 27: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/27.jpg)
27Local Education Agency (S2) Academic Outcomes
![Page 28: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/28.jpg)
28LEA 1: Comparison data (1)Percent of Students Earning HS CC
State Average
LEA Average
![Page 29: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/29.jpg)
29
Comparison Data (2): Context
Per Pupil Expenditure
Earning HS Course Credits
FT teachers
Entering below
grade level % LEP
Facility A $500 70% 5 65% 25%
Facility B $450 40% 5 10% 40%
Facility C $550 20% 5 91% 70%
Facility D $600 33% 5 50% 30%
![Page 30: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/30.jpg)
30
Longitudinal data: more context
![Page 31: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/31.jpg)
31
Do you know enough?
Sometimes, the data will lead to more questions and a need for more information…
Compare to other LEA’s facilities Use student-level data and disaggregate Look at monitoring information and applications Collect additional information-surveys, interviews
*Keep data quality in mind
![Page 32: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/32.jpg)
32
Step 3: Implement improvement plan
Implement new programming, change, etc.
Set benchmarks, performance targets− In terms of your priorities, where do you want your
subgrantees and facilities to be in one year? Two years? Three years?
− What performance benchmarks might you set to measure progress along the way?
− How will you know when to target a subgrantee or facility for technical assistance? At what point might you sound the alarm?
![Page 33: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/33.jpg)
33Step 4: Develop processes for reviewing data
Keep using it! Monitor change and compare against benchmarks
Review data in real time
Share it and discuss it
![Page 34: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/34.jpg)
34
Keep in mind
Data use is not easy* Data should be a flashlight, not a hammer* Change takes time-set realistic goals “No outcome” can be a useful finding Aggregated data can usually be shared
*Source: Data Quality Campaign
![Page 35: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/35.jpg)
35Data Capacity Exists !(Data Quality Campaign, 2011 Report)
10 Essential Elements of Longitudinal Data Systems # States
A unique student identifier 52
Student-level enrollment, demographic, and program participation information 52
The ability to match individual students’ test records from year to year to measure academic growth
52
Information on untested students and the reasons why they were not tested 51
A teacher identifier system with the ability to match teachers to students 44
Student-level transcript data, including information on courses completed and grades earned
41
Student-level college readiness test scores 50
Student-level graduation and dropout data 52
The ability to match student records between the P–12 and postsecondary systems 49
A state data audit system assessing data quality, validity, and reliability 52
![Page 36: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/36.jpg)
36
1. Link State K-12 data systems with early learning, postsecondary education, workforce, social services, and other critical agencies. 11
2. Create stable, sustained support for robust state longitudinal data systems. 27
3. Develop governance structures to guide data collection, sharing, and use. 36
4. Build state data repositories that integrate student, staff, financial, and facility data. 445. Implement systems to provide all stakeholders with timely access to the information they need while protecting student privacy. 2
6. Create progress reports with individual student data that provide information educators, parents, and students can use to improve student performance. 297. Create reports that include longitudinal statistics on school systems and groups of students to guide school-, district-, and state-level improvement efforts. 368. Develop a purposeful research agenda and collaborate with universities, researchers, and intermediary groups to explore the data for useful information. 319. Implement policies and promote practices, including professional development and credentialing, to ensure that educators know how to access, analyze, and use data appropriately. 310. Promote strategies to raise awareness of available data and ensure that all key stakeholders, including state policymakers, know how to access, analyze, and use the information. 23
Next Step: Data Use (DQC-2011)
![Page 37: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/37.jpg)
37
Accessible Data – N or D Related
Title I, Part D Data ED Data Express:
www.eddataexpress.ed.gov NDTAC State Fast Facts Pages:
http://data.neglected-delinquent.org/index.php?id=01 Title I, Part D, Annual Report:
www.neglected-delinquent.org/nd/data/annual_report.asp
Civil Rights Data Collection (district and school)http://ocrdata.ed.gov/
![Page 38: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/38.jpg)
38
Accessible Data – N or D Related
OSEP Data Collectionhttps://www.ideadata.org/default.asp
Youth Behavior Survey (CDC) http://www.cdc.gov/healthyyouth/yrbs/index.htm
OJJDP Juvenile Justice Surveys /Data Bookhttp://www.ojjdp.gov/ojstatbb/
![Page 39: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/39.jpg)
39
Resources
NDTAC reporting and evaluation resources: http://www.neglected-delinquent.org/nd/topics/index2.php?id=9
Data Quality Campaign: www.dataqualitycampaign.org Data for Action 2011—Empower With Data
![Page 40: Using Data for Program Quality Improvement Stephanie Lampron, Deputy Director.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f065503460f94c1bae3/html5/thumbnails/40.jpg)
40
Questions?
Stephanie Lampron
NDTAC Deputy Director
202-403-6822
NDTAC Data Team Dory Seidel: [email protected] Liann Seiter: [email protected]