Using High-Quality Data to Monitor Student Performance Mark Baird, Ph.D. Division of Career and...
-
Upload
eustace-white -
Category
Documents
-
view
214 -
download
0
Transcript of Using High-Quality Data to Monitor Student Performance Mark Baird, Ph.D. Division of Career and...
Using High-Quality Data to Monitor
Student Performance
Mark Baird, Ph.D.Division of Career and Adult
Education
AECP Leadership InstituteDecember 5, 2012
The Impact of Fee and Residency Policy Changes on 2011-12 AGE Enrollment
Compared to Previous Year
School DistrictsHeadcount: -33.9%FTE: -31.6%
Florida College SystemHeadcount: -31.0%FTE: -29.4%
School District Enrollment (Headcount) Changes
Program 2010-11 2011-12 DifferenceAcademic Skills for Adult ESOL Learners 2,545 1,849 -27.3%Adult Basic Education (ABE) 78,914 51,802 -34.4%Adult English for Speakers of Other Languages (ESOL) 94,402 59,215 -37.3%Adult General Education for Adults with Disablitites Educational Plan 809 994 22.9%Adult High School 13,074 10,545 -19.3%Adult High School Co-Enrolled 57,070 36,134 -36.7%Applied Academics for Adult Education 11,973 8,814 -26.4%Citizenship 6,874 6,855 -0.3%English Literacy for Career and Technical Education (ELCATE) 4,386 6,326 44.2%General Educational Development (GED) 31,722 21,051 -33.6%Literacy Skills for Adult ESOL Learners 4,882 2,411 -50.6%Pre-Applied Academics for Adult Education 2,386 1,781 -25.4%Pre-General Education Development (GED) 5,310 3,806 -28.3%Workplace Readiness Skills for Adult ESOL Learners 562 577 2.7%
Florida College System Enrollment (Headcount) Changes
Course 2010-11 2011-12 DifferenceAdult Basic 31,396 22,123 -29.5%Adult Secondary 6,436 5,034 -21.8%EAP Literacy 14,916 10,070 -32.5%EAP Vocational Prep. 3,883 3,333 -14.2%GED Prep. 10,646 6,944 -34.8%Vocational Prep. 5,018 2,398 -52.2%
Enrollment Decline More Dramatic When Summer Excluded (District Data)
All Terms Winter/Spring Terms
2010-11 268,663 229,5072011-12 177,559 132,840Decline 91,104 96,667Percent Decline 33.9% 42.1%
Early Indication That Completion Rate Increased
District AGE Program2010-11, No Summer 2011-12, No Summer
Completion to Headcount Ratio
Completion to Headcount Ratio
Adult Basic Education (ABE) 0.52 0.65Adult High School 2.25 2.49General Educational Development (GED) 1.23 1.27Adult English for Speakers of Other Languages (ESOL) 0.41 0.45English Literacy for Career and Technical Education (ELCATE) 0.44 0.29Academic Skills for Adult ESOL Learners 0.23 0.29Workplace Readiness Skills for Adult ESOL Learners 0.53 0.47Citizenship 0.31 0.41Adult General Education for Adults with Disablitites Educational Plan 0.00 0.26Pre-General Education Development (GED) 0.49 0.48Literacy Skills for Adult ESOL Learners 0.40 0.50Pre-Applied Academics for Adult Education 0.32 0.35Applied Academics for Adult Education 0.55 0.66TOTAL 0.63 0.72
The Central Questions
How do you ensure the quality of your data?
How do you use this high-quality data to improve your programs?
Questions of Program Quality Can you answer these questions
without using anecdotal evidence? How good are your programs? How good are your teachers? How much are your students learning? Are your programs a good investment for
Florida taxpayers? Are your programs as effective and efficient
as they should be?
Four Key Areas of Accountability Measurement
EnrollmentProgressCompletionOutcomes
The Necessity of Locally Generated Reports Performance reports from the state will
ALWAYS be significantly lagged Not all useful data are reported to the
state Timely local response to student
performance data feedback is critical The timeliest and most granular
performance data will always come from local data systems
Local Reports Local data needs to be updated on a
regular basis (daily if possible) Store data in a back-end database Use front-end tool to slice and dice data
(e.g. Access, Excel, SPSS) Create reports that can be distributed
to stakeholders Liberate your data!!!
The Key Feedback Loop
Classrooms & Testing Centers
Local Program Office
If this is not flowing, you have a
problem
The Data Roach Motel Problem
Data go in, but it doesn’t get out.
The Citrus Solution
Classrooms & Testing Centers
Local Program Office
Program Database
NRS Accountability Reports
Your check engine Light
What You Can Get Out of NRS Data
ABE & ESOL: Percentage of students who complete a functioning level
ASE low: Percentage of students completing courses or GED® subtests.
ASE high: Percentage of students earning diploma
Average instructional hours Comparisons of above between post-
tested and non-post-tested students.
What do WE look at? Percentage of students reported as
post-tested Percentage of students earning LCP Average LCPs per student Average instructional hours per LCP Percentage of ASE students earning
diplomas Transition of diploma earners to
postsecondary education and employment
Transition of ABE High completers to ASE or diploma
Transition of ESOL students to ABE, ASE, diploma or postsecondary
Percentage of GED® Prep students taking all five subtests and of those, percentage earning diploma
DOE Reports Provided to District and College AGE Directors
LEA-Level NRS Report Student Progression (NRS) Student Instructional Hours (NRS) Student Demographics Student Performance Student Transition
New State Data Tools
Florida is blessed with copious data Complex systems of data collection and
management are in place Next step: Getting data into the hands
of administrators and teachers Reports and tools need to be intuitive
and easy to use; if not, formal training required
Statewide Longitudinal Data System (SLDS) Projects
Upgrade state data systems and consolidate silos into a comprehensive Education Data Warehouse v 2.0.
Customizable report-building tools and dashboards for stakeholders
Access to state data via single sign-on gateway (“one-stop shopping”).
Data mining tool for FLDOE researchers Minimum standards for local data systems, a forum for
information exchange, and financial support to small and rural districts.
Statewide unique identifier to all students and staff. State research agenda and automated delivery of
data to researchers.
Local Student Data Management Requirements in AEFLA Grant AEFLA grantees must
Collect data from instructional and testing sites on a monthly basis at a minimum and store the data in a relational format
Produce monthly reports and make them available to instructional and testing sites for review and correction
Produce monthly reports, as requested locally, for program improvement and monitoring
Use Your Local Data: Examples of Local Report Content Instructional site administrators should have
access to timely classroom-level data on: Attendance (e.g. average monthly) Completions (e.g. completion rate) Contact hours between test administrations Ratio of headcount and contact hours to
completions Disaggregation by race, gender, and other
relevant variables, e.g. day/evening classes
Test Data
Beginning in 2010-11, the state required school districts and colleges to report student-level literacy test data for adult education students
Use the data for your own analysis by comparing pre- and post-test scale scores
Limitation of Measurement by Functioning Level
Time
Tes
t S
core
s
Level 2 Floor
Level 1 FloorPre-test
No learning gain recorded despite growth
Learning Growth
Curve
Post-testIs this a better teacher?
LOOK OUT! Comprehensive Courses: AKA The BLOB!
Impossible to determine the success rate of students in comprehensive ABE and GED courses by skill/subject area and functioning level
NRS Changes for 2012-13 Adult student goal will no longer be used
to identify follow-up cohorts (Table 5) Enter employment and job retention based
on employment status at entry Enter postsecondary cohort will be all
diploma earners Obtain secondary credential
Adult high school: Number of ASE high students who earn diploma
GED: Number of students taking all subtests who earn GED (pass rate)
NRS Changes for 2012-13
Participant Status (Table 6) Highest degree or level of schooling
completed U.S.-based schooling Non-U.S.-based schooling
Personnel (Table 7) Teachers’ years of experience Teacher certification
2011-12 Data Reporting Approximately half of all districts had extensive
problems with their data on the first load attempt Two districts were required to submit supplemental
files to correct instructional hours because of pervasive problems
Districts with over 1% of AGE course records invalid Supplemental files accepted to validate records
and include instructional hours in funding calculations
Work on 2011-12 WDIS data ended on October 15, 52 work days after the close date.
Numerous reporting issues on the college side
Data Quality Issues
Problems we saw in 2011-12 WDIS reporting Invalidated course records Over-reporting of instructional hours Under-reporting of instructional hours Large changes in enrollment or average
instructional hours compared to previous year
Some district reporting staff are Not reviewing data before it is submitted Not reviewing feedback reports on
submitted data to investigate and fix errors Files of student records flagged with validation
errors are available for download and analysis
State MIS staff do not have the time to fix local data problems or perform analysis that local staff should handle
WDIS Submission Periods, 2012-13
Survey Period Opens Required Load Period Closes
F/G Summer & Prelim Fall
September 4, 2012
September 13, 2012
October 18, 2012
W/X Fall & Prelim Winter
January 7, 2013 February 7, 2013 March 7, 2013
S June 3, 2013 July 3, 2013 July 11, 2013
S Update Window
July 15, 2013 N/A August 1, 2013
College Data Submission Periods, 2012-13
Submission Period
Period Opens Required Load Period Closes
Summer End/Fall Beginning
August 27, 2012September 24, 2012
October 8, 2012
Fall End/Winter-Spring Beginning
January 7, 2013 February 4, 2013 March 4, 2013
Winter-Spring End
April 22, 2013 May 13, 2013 June 17, 2013
The time between the required load date and the close of the submission period allows for corrections to be made and data to be resubmitted
Some are missing the load deadline and loading data at the close of the submission period, allowing no time for corrections.
Developing a Local Data Quality Assurance System Identify your Reports Coordinator Build reporting timelines into your planning
calendars Establish with Reports Coordinator a data review
committee that includes program, data, and budget staff Determine what pre-load reports the committee should
review Ensure that the committee has access to validation and
mid-survey reports provided by FDOE showing data you have submitted
For each survey, set up times for the committee to meet and review data twice: Before the data are first loaded After data are loaded but before the close of the survey
The Submission Review Process
1. Review data2. Fix problems3. Load data4. Review edit and validation reports5. Review mid-survey reports6. Fix problems7. Certify data before deadline
Pre-Load
Post-Load
Pre-Load – Review Local Reports You should have a set of local reports to
review, by program, before data are loaded into WDIS
If your system is incapable of producing pre-load data reports, this is an unacceptable situation
Ideally your system should run all WDIS validation edits locally BEFORE the data are loaded CCTCMIS can provide COBOL code that runs the
data through the edits
Pre-Load – What to Look for In Local Reports
If your system runs edits, number and percentage of valid records
Average instructional hours by program; school and program
LCPs reported by program; school and program
Another Fine Mess
Not reviewing your data before you load it is like not scraping out the lasagna dish before putting it in the dishwasher.
Post-Load – Review Reports from Mainframe and Clean Data
When your data are loaded, the system generates reports upon request How many records were accepted, rejected
with critical errors, and flagged with non-critical errors
Lists of student records with error codes This information should be used to
clean up submitted data
The mainframe verification reports are accessible online, but are secured requiring login and password information
Your Reports Coordinator must have access to these reports– it is possible that is not the case
Post-Load - Review Mid-Survey Reports to Detect Anomalies
CCTCMIS produces a set of mid-survey WDIS reports that are useful for detecting anomalies in your data These reports are available on the CCTCMIS
restricted access website: http://www.fldoehub.org/CCTCMIS/Pages/default.aspx
College reports on enrollments, FTE, and completions available on mainframe
Your Reports Coordinator must have a username and password from CCTCMIS for authorized login
Post-Load – Key WDIS Mid-Survey Reports
Enrollment and Instructional Hour Comparison with Previous Year 01 – By District 03 – By Program 05 – By School 07 – By School and Program 07A – By School, Program, and Course
11 - Valid Course Records
Completer Counts 14 – By District 15 – By Program 16 – By School 17 – By School and Program
State MIS Advisory Committees The Workforce Education District Data Advisory
Committee (WEDDAC) is an group comprising representatives from school district workforce education data reporting units
The Management Information Advisory Task Force (MISATFOR) is the corresponding college organization
The groups meet jointly with FDOE staff approximately three times a year to discuss data issues, including changes to the state’s databases
Make sure your district/college is represented!
Consequences of Poor Data QualityFundingAccountabilityCredibilityAudit Risk
ContactsAccountability & Reporting
[email protected] , [email protected] , 850-245-9060
State BudgetDistricts – [email protected] ,
850-245-9002State Colleges – [email protected] ,
850-245-9764Federal Grants
Your regional program manager