Using Data to Measure Outcomes
-
Upload
ingrid-small -
Category
Documents
-
view
33 -
download
1
description
Transcript of Using Data to Measure Outcomes
Using Data to Using Data to Measure OutcomesMeasure Outcomes
Student Tracking and Using Data Effectively
Geoff Zimmerman
2004 NTPN – Minneapolis, MN
Impact?
What impact are you having on the students and schools that you are serving?
Basic Question:
Goals for Today
1. Big Picture: Impact/Outcomes
2. Nuts & Bolts: Data Collection
3. Tech Prep in Ohio: Measuring Success
4. Demonstration of Database Application
5. The Basics – Designing Your Own
Are you Data-Driven?
In business and industry, data are king. Information about customers, inventory, sales, rates of return and employee turnover is crucial. Good data determines success or failure.
= Educational Accountability
Why Collect Data?
Measure progress
Make program decisions
Perkins driven – shows accountability
Provides hard data to state executive and legislators when making funding decisions
Accountability: Guaranteeing Results
Flexibility: Local Control for Local Challenges
Research-Based Reforms: Proven Methods with Proven Results
Parental Options: Choices for Parents, Hope for Kids
Four Reform PrinciplesFour Reform Principles
No Child Left Behind
Perkins III (1) Definition Section 202(a)(3) of Perkins III states that a "tech-prep program" means a
program of study that: combines at a minimum 2 years of secondary education (as determined
under State law) with a minimum of 2 years of postsecondary education in a nonduplicative, sequential course of study;
integrates academic, and vocational and technical, instruction, and utilize work-based and worksite learning where appropriate and available;
provides technical preparation in a career field such as engineering technology, applied science, a mechanical, industrial, or practical art or trade, agriculture, health occupations, business, or applied economics;
builds student competence in mathematics, science, reading, writing, communications, economics, and workplace skills through applied, contextual academics, and integrated instruction, in a coherent sequence of courses;
leads to an associate or a baccalaureate degree or a postsecondary certificate in a specific career field; and
leads to placement in appropriate employment or to further education. An allowable tech prep program must meet the terms of this definition.
Are your programs making a difference?
Beginning Questions How many students are enrolled in a Tech Prep program? What is the college transition rate for Tech Prep Students? How
many are persisting in college? What are the demographic characteristics of Tech Prep
students? To what extent are Tech Prep students prepared for college?
Measuring Outcomes
Why do you exist? What conditions need to change for this
target group and why?
Describe your program – what does it do and what does it accomplish?
Source: United Way of Greater St. Louis Guide for Measuring and Reporting Outcomes
Measuring Outcomes
What is your program designed to do? Who is the target of the change? Who
benefits from your services? What information do you need to compile
about participants to ensure delivery? What measurable changes will occur?
(cont.)
Source: United Way of Greater St. Louis Guide for Measuring and Reporting Outcomes
Describing what you do
Inputs – resources used to implement the program.
Activities – actions that define what is done, when, where, how, how often, and for whom.
Outputs – the direct products of the program activities. Measure of the amount and volume of work performed to implement the program.
Source: United Way of Greater St. Louis Guide for Measuring and Reporting Outcomes
Describing your intended results
Outcomes – the results the program aspires to accomplish; they specify immediate, intermediate, and long-term changes among students served by the program
Indicators – measurable data about benefits for and changes in the target group that is collected at the individual level to show an outcome has been achieved
Impact – what the program aspires to accomplish over a long period of time
Source: United Way of Greater St. Louis Guide for Measuring and Reporting Outcomes
Outputs vs. Outcomes
Output = what you do
Outcome = what changes because of what you do
-refining selections as a result of college fair or visitas opposed to x number of students attending the college fair
-completing and submitting college applications following advising sessionas opposed to advising x number of students
-submitting a FAFSA on timeas opposed to giving the FAFSA form to x number of students
-persisting in college as opposed to enrolling in college
Logic Model
Inputs Activities Outputs
Outcomes
Impact
Logic Model (alternative)
Activities
Outcomes
Long-Term Impact
Activities
Outcomes
Activities
Outcomes
Uses of Outcome Information
“A key element in effective use of outcome information is to continually ask why differences occur and then assess whether actions can be taken to improve results.” (p.15)
Hatry, H., Morley, E., Rossman, S., and Wholey, J. (2003). How Federal Programs Use Outcome Information: Opportunities for Federal Managers. IBM Endowment for the Business of Government. Online: http://www.businessofgovernment.org/pdfs/HatryReport.pdf
In other words…
In other words outcome information Tells us what is happening So we can ask why it is happening And fix it if it needs to be fixed Celebrate it and tell everyone if it works
Uses of Outcome Information
How are you using it? Triggering corrective action Identifying and encouraging “best practices” Motivating Planning and Budgeting Reporting to Key leaders
*Hatry, et.al, How Federal Programs Use Outcome Information: Opportunities for Federal Managers
Quality is not just a chart, or a standard,or a specification—it’s a state of mind, a commitment, a responsibility, a spirit.It’s a way of doing, being, living.
—Don Galer
Data Collection
“Insanity: the belief that one can get different results by doing the same thing.”
-Albert Einstein
How to Collect?
•Student enrollment form
•Student list report of file from schools
•Surveys – students, parents, teachers, etc.
•Focus groups, interviews
•Program site review process
U.S. Census Bureau, State Departments of Education, National Center for Education Statistics
Primary Data Sources
Secondary Data Sources
What to Collect
Determine what data to collect Demographics Participation in activities College aspirations Academic preparation Persistence in college Remediation in college Feelings about the program Perceptions of education and work
2 year A.S. 1-2 year Tech 4 year degree don't know
How much additional education do you think you will need?
0
10
20
30
40
50
60F
req
uen
cy
How much additional education do you think you will need?
Survey Results
Source: Greater Cincinnati Tech Prep Consortium, May 2004
Very Helpful Somewhat Helpful Not Very Helpful Not at all helpful
How much has this program helped you in feeling your time in school is more worthwhile?
0
10
20
30
40
50
60F
req
uen
cy
How much has this program helped you in feeling your time in school is more worthwhile?
Survey Results
Source: Greater Cincinnati Tech Prep Consortium, May 2004
Very Helpful Somewhat helpful Not Very Helpful Not at all Helpful
How much has this program helped you in improving your grades?
0
10
20
30
40
50F
req
uen
cy
How much has this program helped you in improving your grades?
Survey Results
Source: Greater Cincinnati Tech Prep Consortium, May 2004
Very Helpful Somewhat Helpful Not Very Helpful Not at all Helpful
How much has this program helped you in knowing what college associate degree programs are available in your
program area?
0
10
20
30
40
50F
req
uen
cy
How much has this program helped you in knowing what college associate degree programs are available in your
program area?
Survey Results
Source: Greater Cincinnati Tech Prep Consortium, May 2004
Very Helpful Somewhat Helpful Not Very Helpful Not at all Helpful?
How much has this program helped you in learning more about your specific technology area?
0
10
20
30
40
50
60F
req
uen
cy
How much has this program helped you in learning more about your specific technology area?
Survey Results
Source: Greater Cincinnati Tech Prep Consortium, May 2004
Performance Measure Question Sample Responses
Articulated credits Did you have the opportunity to earn articulated credits while in
high school?
-We did not earn any.-I had 12 but the four-year college would only use them as electives.-I had AP credits in English and math, but none in Tech Prep.
Rigorous academic curriculum Did your academic curriculum prepare you for college?
-Some courses were not challenging. -I did an independent study in math as my class was behind me. -Locking you into the pathway allows for no deviation based on your ability.
Market share How can we attract more students into Tech Prep programs?
- Use Tech Prep students as program ambassadors to inform other students and parents about program oportunitieis-Start earlier, in the 7th and 8th grades.
Focus Group Results
Source: Greater Cincinnati Tech Prep Consortium, September 2003
Percent of population 25 years and over 100.0 100.0 100.0
Less than 5th grade 0.9 0.0 1.5
5th to 8th grade 6.3 6.3 6.3
9th to 12th grade, no diploma 21.1 21.3 20.9
High school graduate (incl. equivalency) 42.8 40.8 44.4
Some college credit, less than 1 year 3.5 3.9 3.1
1 or more years of college, no degree 10.9 8.9 12.5
Associate degree 3.4 3.1 3.7
Bachelor's degree 7.7 10.5 5.5
Master's degree 1.5 3.4 0.0
Professional degree 1.8 1.3 2.1
Doctorate degree 0.3 0.6 0.0
Percent high school graduate or higher 71.8 72.4 71.2
Percent bachelor's degree or higher 11.3 15.7 7.6
Educational Attainment, Bethel, Ohio
QT-P20. Educational Attainment by Sex: 2000Data Set: Census 2000 Summary File 3 (SF 3) - Sample Data http://factfinder.census.gov
Tech Prep in Ohio
Ohio College Tech Prep combines college preparatory academics and advanced career-technical education into a seamless program from high school to college to high-skill, high-wage careers.
Innovative Local Delivery Models
• CTP programs on college campuses• CTP programs in industry setting• Industry mentorships• Dual Enrollment and Transcripted college credit options
• College faculty team-teaching with secondary instructors
College TP goals and Benchmarks
Ohio Tech Prep Goals (1998)1. Expand CTP enrollment to meet
Ohio’s critical workforce and economic needs
2. Provide a CTP system that reflects the requirements for success in college and in high-wage, high-skill technical careers.
3. Maximize student options through the integration of high school, associate degree and baccalaureate degree pathways to achieve a seamless, non-duplicative system.
4. Engage the active support and involvement of all College Tech Prep stakeholders.
Performance BenchmarksExpand enrollment: 15% of all 11th and 12th graders
Enter college remediation free: 10% higher rate than traditional college population
Transition to college: 66%
Under-represented groups: 25%➲
➲
➲
➲
3. Building the Secondary Pipeline
3109
1458
9532
11419
6740
4995
607211
11530
12882
0 0 57 212
513 835 1288
1673
35305176
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
12000
13000
14000
93-94 94-95 95-96 96-97 97-98 98-99 99-00 '00-01 '01-02 '02-03
High School
College
Annual Enrollment
Student Tracking and Analysis though HEI
Ohio Board of Regents Higher Education Information (HEI)
Student Tracking System
How is this done? Ohio has a comprehensive postsecondary
data collection system in place - Higher Education Information System (HEI)
Available to all public two & four year institutions.
Limited access to private colleges
How HEI Works
Tracks students by SSN Follows student mobility in higher
education. Tracks by campus type
State Community College & Community Colleges Technical Colleges University Branch or Main Campuses
Reports on remediation levels and persistence
HEI Enrollment Data SubmissionsTech Prep Consortium Tracking (TC) File
Revised May 1, 2002File Description:The Tech Prep Consortium Tracking (TC) File contains one record for each student enrolled in a Tech Prep consortium.
Submission Schedule:The TC file can be submitted anytime throughout the year.
Field Names Field Attributes and Procedures Data Format
Consortium Code Enter the consortium code from the consortium code verification table.
4 Alphabetic Characters, Columns 1-4
Student Identifier Enter the federally assigned Social Security Number (SSN) whenever possible. If the SSN is unavailable, enter another identifier which uniquely relates to this student. If the Student Identifier is not a SSN, it can be entered in the TC file, but the student’s enrollments cannot be tracked.
9 Alphanumeric Characters,
Columns 5-13
Institution Assigned Identifier Switch
Enter “Y” if the Student Identifier is not a federally assigned Social Security Number (SSN). Otherwise enter “N”.
1 Alphabetic Character, Column 14
Data Fields:
Field Names Field Attributes and Procedures Data Format
Program Code Enter the code of the program of study in which the student was enrolled from the Program Code verification table.
6 Numeric Characters, Columns 15-20
Career Center Code
Enter the code of the Career Center or Tech Center from which the student enrolled in Tech Prep from the IRN verification table. If there is no Career Center or Tech Center associated with this student, or to report the association of a compact or collaborative, enter 000000.
6 Numeric Characters, Columns 21-26
High School Code Enter the code of the high school from which the student graduated or will graduate from, from the IRN verification table.
6 Numeric Characters, Columns 27-32
Site of Delivery Enter the code of the site of delivery from which the student received instruction from the IRN verification table. If the site of delivery was a hospital or business, enter 000000. If the site of delivery was a college or university in Ohio, enter a code from the IPEDS/Institutional code verification table.
6 Numeric Characters, Columns 33-38
Graduation Year Enter the year this student graduated, or will graduate from high school.
4 Numeric Characters, Columns 39-42
Delete Switch Enter “Y” if the record is to be deleted from the database. Otherwise, enter “N”.
1 Alphabetic Character, Column 43
Data Fields (continued):
HEI Enrollment Data SubmissionsTech Prep Consortium Tracking (TC) File
Link to HEI – 16 digit code Consortium identifier Student identifier (ss#) Program Code Career Center Code (IRN) Delivery Site Code (IRN) High School Code (IRN) Graduation Year
(CNCI999990153Y11T0110514820065930065932004N)
Internal Data Tracking System
Student Enrollment Forms (paper filing) Microsoft Excel, Lotus Microsoft Access College databases for transition enrollment,
remediation
What Data are Gathered
Consulted and sought input from consortia directors on four major topics. Tech Prep Completion Remediation Transition and Persistence Underrepresented Populations
Improved Remediation Rates
Tech Prep vs. Non-Tech Prep Remediation Rates(18-19 year olds)
45 %
0
400
800
1200
1600
2000
Tech Prep Students
Non-Remedial
RemedialStudents
57.4 %
0
5000
10000
15000
20000
25000
Non-Tech Prep Students
Population are 18-19 year old freshman enrolled at community / technical colleges or university branch campuses.1604 students total 21337 students total
Remediation
Number and Percent Taking Remedial Coursework in AY 2002 (data as of 8/26/03)
Student type Enrollees # math % math # eng % eng # math or eng
% math or eng
# math and eng
% math and eng
Non tech prep 14,295 7,458 52% 5,123 36% 8,894 62% 3,687 26%
Tech prep 2,251 876 39% 644 29% 1,102 49% 418 19%
Number and Percent Taking Remedial Coursework in AY 2002 (data as of 8/26/03)
Student type Enrollees # math % math # eng % eng # math or eng
% math or eng
# math and eng
% math and eng
Non tech prep 41,701 9,118 22% 6,199 15% 12,028 27% 3,289 8%
Tech prep 1,546 371 24% 243 17% 513 33% 121 8%
Two – Year Campuses 2003-04
Four - Year Campuses 2003-04
(cont.)
PersistenceFirst-time, Full-time, degree-seeking Undergraduate Students age 19 and under in Autumn Term. Number and percent persisting to Autumn of next year, by type (tech prep vs. non tech prep). Persisting includes enrollment at an Ohio public institution, or private institution if full-time and receiving a Student Choice Grant, or proprietary institution if receiving a Workforce Development Grant. Data as of 8/27/2003.
* First Time Full Time Degree Seeking Undergraduates
Year Campus Type Tech Prep * FTFTDSUG # persisting % persisting
2000 University Branch N 5,709 4,223 74.0%
2000 University Branch Y 108 83 76.9%
2000 University Main Campus N 31,348 27,109 86.5%
2000 University Main Campus Y 206 156 75.7%
2001 Community College N 2,960 2,000 67.6%
2001 Community College Y 135 96 71.1%
2001 State Comm. College N 2,774 1,741 62.8%
2001 State Comm. College Y 122 65 53.3%
2001 Technical College N 1,654 1,049 63.4%
2001 Technical College Y 101 64 63.4%
2001 University Branch N 5,648 4,258 75.4%
2001 University Branch Y 120 86 71.7%
2001 University Main Campus N 32,160 27,720 86.2%
2001 University Main Campus Y 412 337 81.8%
* First Time Full Time Degree Seeking Undergraduates
New Data to be Collected . .
Graduate’s Age upon Graduation
Non tech prep students <24 years
Tech prep students <24 years
Non tech prep students 24+ years
Tech prep students 24+ years
Discipline Subject # of Graduates
4th Year Salary
# of graduates
4th year Salary
# or graduates
4th year Salary
# of graduates
4th year Salary
Education
(3)
Sports & Recreation
Teaching
Engineering
(4)
Engineering Technology
Associate Degree Graduates: Fourth Year Following Graduation by DisciplineAnnualized Full-Time* Salaries (in thousands)
• Post-program Placement
Demo
Consortium office database for tracking students, programs, articulation agreements, and contacts
Contact information:
Geoff ZimmermanGreater Cincinnati Tech Prep ConsortiumUniversity of [email protected]