Initial Research Findings from Arkansas’ Statewide Longitudinal Data System:

Using Standards-Based Research to Solve Real Education Problems

Jake Walker, PhD, Project Director Arkansas Research CenterGreg Nadeau, Director, State Core Model, Public Consulting Group

2012 MIS Conference 1

The CEDS Data Model Process

2

A foundation for standards convergence

CEDS Domain Entity Schema (DES)

Domain EntityEL EL ChildEL EL EnrollmentEL EL AssessmentEL EL ProgramEL EL GroupEL EL StaffEL EL Parent

Domain EntityK12 K12 StudentK12 K12 IncidentK12 K12 ParentK12 K12 StaffK12 K12 Staff AssignmentK12 K12 SectionK12 K-12 CourseK12 K12 Migrant FacilityK12 K12 ProgramK12 K12 SchoolK12 K12 DistrictK12 State Education AgencyK12 AssessmentK12 Learning Standard

Domain EntityPS PS StudentPS PS AssessmentPS PS StaffPS PS SectionPS PS ProgramPS PS Defined ProgramPS PS Teacher ProgramPS PS CampusPS PS SchoolPS PS Institution

CEDS NDS Conceptual Model

Organizations People

Assessment Standards[Content]

Roles

Time

CEDS Logical Model (NDS): High Level

6

7

Parsing NCES Forum Questions with a Standard Syntax

Using CEDS for Analysis# Logic Statement1 The purpose of this artifact is to provide a multi-criterion decision framework to improve Secondary Data Analysis

from P-20 Longitudinal Data Systems. It assumes a set of education and workforce data that is common across the 50 states.

2 In this artifact we will lay out a P-20 Standard Analytic Model (P-20 SAM) built off of the new CEDS 2.0 Data Model.

3 The P-20 SAM is a framework for quantitative research and performance management utilizing valid metrics based on standard data definitions and relationships. It can be used as a logical framework for dimensional models.

4 Valid metrics are aggregate measures (counts, averages, ratios) that have been demonstrated to correlate with educational indicators. For example Student Growth Percentile (SGP) is a measure that correlates with K12 quality learning and Degree Production Rate (DPR) is a measure that correlates with postsecondary effectiveness. A list of P-20 Metrics is provided with a standard definition with transformation rules tied directly to standard CEDS DES element definitions and NDS tables and fields.

5 A list of P-20 Metrics is provided with a standard definition with transformation rules tied directly to standard CEDS DES element definitions and NDS tables and fields.

6 The P-20 SAM provides Standard Syntax for expressing research questions and performance reports. This syntax must specify the metric, entity/ies, and daterange/s to be considered.

7 Metrics are often expressed as rates. Rates express the ratio between a sub-group count (little n or numerator) and a general population count (big N or denominator).

8 Application of a metric to a given data set must consider the quality and quantity (n-size) of the data as well as the sensitivity of the metric to determine the overall reliability of the measure.

A Framework for Using CEDS for Analysis and Performance Management

Using the Framework• Step 1: Framing the Issue

• The Question • The Audience• The Audience’s Focuses• The Approach

• Step 2: Selecting the Right Metrics• Data Sources • Data Matching• Variables Used

• Step 3: Constraining the Metrics• Output

• Step 4: Data Use• Wages by Degree Level by Major• UAMS Study 10

The Question

• What is the value of public higher education in the state of Arkansas?

11

The Audience

• State Level Policy Makers• Administrators of Public Institutions of

Higher Education• Students and Families

12

The Audience Focuses

• Policy and Funding Issues• Program Planning and Staffing Issues• College Outcomes for Life

13

The Approach

Original Report• What percentage of

Arkansas residents and non-residents who graduate from public colleges and universities remain in Arkansas?

• How does level of education impact salary?

• What is the average salary by field of study and level of degree of Arkansas graduates?

New Version• What are the employment

outcomes of graduates from 2002 to 2009 as of 2010 in the state?

• What percentage of Arkansas residents and non-residents who graduate from public colleges and universities remain in Arkansas either in the workforce or enrolled in the higher education system?

• How does level of education impact salary?

• What is the median salary by field of study and level of degree of Arkansas graduates?

• In which industry areas are Arkansas graduates employed after they finish college?

14

Variables UsedOriginal Report

• SSN • Graduation Year• Degree Level• Degree Area• Residency Status• Employment Status• Annual Wages at Year 1• Annual Wages at Year 5 • Residency Status

New Version• SSN• DOB• First Name• Last Name• Graduation Date (MM/YYYY)• Degree Level• Degree Area• Residency Status• Enrollment Status• Employment Status• Average Quarters Employed• Annual Wages at Year 1 • Annual Wages at Year 2• Annual Wages at Year 5• NAICS Code

15

ADH

E

ADH

E

ADW

S

ADW

S

Data SourcesOriginal Report

• Arkansas Department of Higher Education

• Arkansas Department of Workforce Services

New Version

• Arkansas Department of Higher Education

• Arkansas Department of Workforce Services

• National Student ClearingHouse

16

Data MatchingOriginal Report

• SSN from ADHE matched to SSN from ADWS

New Version

• TrustEd Broker System• SSN• DOB• First Name• Last Name

• Uses multiple pieces of information about a person to determine if they match.

• Personally identifiable information is removed and replaced with a non-traceable id.

17

• ~55,000,000 records for 10 years, 2,938,718 unique SSNs, no DOBs, inconsistent naming standards.

• 7,865 SSNs used by two or more people, for a total of 18,278 different individuals. Those would be combined incomes and treated as the same person if SSN was the primary key.

• The same person has two or more SSNs (because of a typo/transposition) 13,373 times. There would be 13,373 additional (non-existent) people with separate incomes if SSN was the primary key.

Identity Resolution (Wage Data)

A trusted broker maintains a cross reference table, encoding the identifiers for various agencies and for various representations of the entities.

Trusted Broker

Bob Smith P0236 Robert Smith K4297

Agency 1 Agency 2

Internal

Identifier Identity Information

Identifier Agency1

Identifier Agency2

KB5765 Bob Smith, Barton P0236 K4297

KB5765 Robert Smith, Barton P0236 K4297

KB5765 Bob Smith, Wilson P0236 K4297

Dual Database: Regulatory Compliance and Privacy

Identity Information: Agency ID Generated

Information of Interest: Stored With Agency ID

Knowledge Base:Identity Management Edge Servers:

Shareable Data

De-identified Data

Output Original Report

• Percentage of Graduates Remaining in Arkansas

• Average Salary by Degree Level

• Average Salary by Degree Level by Degree Area/Field of Study

New Version• Percentage of Graduates

Remaining in Arkansas • Median Wages and

Average Quarters Employed by Degree Level

• Median Wages and Average Quarters Employed by Degree Level by Degree Area/Field of Study

• Industry Area Employed in by Degree Level

22

Population of Interest and Entry Time Into the Workforce

• Old Report• Graduates from an academic year would only be counted as

employed if they were present in wage data during the 4th QUARTER AFTER THE END OF THE GRADUATION YEAR.

• New Report• Graduates were classified if they were employed at any point

during the FIRST FOUR FULL QUARTERS AFTER THEIR GRADUATION DATE.

23

Population of Interest and Entry Time Into the Workforce

• Example• Graduates from the academic year 2005-2006• Anyone who graduated from 07/01/2005 to 06/30/2006

• Old Report• 4th Quarter after the graduation year=2nd quarter of 2007

(April-June 2007)• Graduates during July-September of 2005 would not be

classified as employed unless they had wages for the 2nd quarter of 2007 which is actually 7 quarters after they graduated not 4.

24

Population of Interest and Entry Time Into the Workforce

• Example• Graduates from the academic year 2005-2006• Anyone who graduated from 07/01/2005 to 06/30/2006

• New Version• First Full Four Quarters after the graduate date

25

1st Four Full Quarters After Graduation

Grad Date Q1 Q2 Q3 Q4

07/2005 to 09/2005 20054 20061 20062 20063

10/2005 to 12/2005 20061 20062 20063 20064

01/2006 to 03/2006 20062 20063 20064 20071

04/2006 to 06/2006 20063 20064 20071 20072

Remained vs. Employed

26

OLDPercentage Remained

= # Employed/# of Graduates

NEWPercentage Remained

= # Employed Only + # Enrolled Only + # Enrolled and Employed/# Graduates

Original Report

Degree LevelPercentage Employed

Year 5Percentage Employed

Year 12002 2003 2004 2006 2007 2008

Certificate 63% 64% 61% 75% 76% 76%Associate 73% 73% 72% 78% 78% 76%Bachelor 62% 63% 64% 74% 72% 71%Master 62% 60% 58% 67% 68% 66%Doctoral 44% 32% 44% 49% 54% 43%1st Professional 55% 61% 57% 51% 63% 62%

2002 2003 2004 2005 2006 2007 2008 2009

% in AR Year 1 77% 85% 84% 82% 82% 83% 85% 87%

% in AR Year 2 66% 70% 64% 64% 67% 70% 73%

% in AR Year 5 55% 55% 48% 51%

% in AR Year 1 87% 90% 90% 90% 91% 92% 91% 90%

% in AR Year 2 81% 85% 86% 85% 85% 87% 88%

% in AR Year 5 73% 76% 77% 76%

% in AR Year 1 89% 90% 90% 90% 90% 90% 89% 90%

% in AR Year 2 84% 86% 86% 86% 86% 85% 85%

% in AR Year 5 75% 75% 74% 75%

% in AR Year 1 80% 80% 81% 82% 83% 81% 81% 80%

% in AR Year 2 73% 74% 74% 76% 77% 75% 75%

% in AR Year 5 67% 64% 64% 66%

% in AR Year 1 75% 73% 73% 73% 73% 74% 72% 75%

% in AR Year 2 70% 68% 67% 68% 68% 69% 68%

% in AR Year 5 63% 61% 59% 60%

% in AR Year 1 48% 51% 49% 53% 49% 64% 63% 63%

% in AR Year 2 45% 50% 49% 51% 62% 62% 62%

% in AR Year 5 55% 59% 57% 61%

% in AR Year 1 53% 48% 60% 52% 55% 62% 50% 53%

% in AR Year 2 47% 40% 53% 44% 48% 54% 43%

% in AR Year 5 45% 33% 44% 42%

New Version

Technical Certificate

Associate’s

Bachelor's

Master's

Doctoral-1st Professional

Doctoral-Research/

Scholarship

Past Graduates Recent Graduates

Graduation Year

Certificate of Proficiency

Remained vs. Employed

27

OLDPercentage Employed

= # Employed/# of Graduates

NEWPercentage Employed

= # Employed + #Employed and Enrolled/# Graduates

Original Report

Degree LevelPercentage Employed

Year 5Percentage Employed

Year 12002 2003 2004 2006 2007 2008

Certificate 63% 64% 61% 75% 76% 76%Associate 73% 73% 72% 78% 78% 76%Bachelor 62% 63% 64% 74% 72% 71%Master 62% 60% 58% 67% 68% 66%Doctoral 44% 32% 44% 49% 54% 43%1st Professional 55% 61% 57% 51% 63% 62%

2002 2003 2004 2005 2006 2007 2008 2009

% in AR Year 1 72% 81% 81% 78% 77% 78% 76% 76%

% in AR Year 2 63% 67% 62% 62% 65% 67% 69% 75%

% in AR Year 5 53% 54% 47% 49%

% in AR Year 1 81% 84% 87% 86% 86% 87% 84% 81%

% in AR Year 2 78% 83% 84% 83% 83% 84% 83% 80%

% in AR Year 5 71% 74% 74% 73%

% in AR Year 1 80% 79% 80% 79% 81% 80% 78% 77%

% in AR Year 2 76% 78% 78% 78% 77% 77% 76% 76%

% in AR Year 5 72% 73% 71% 72%

% in AR Year 1 75% 77% 77% 78% 79% 77% 76% 76%

% in AR Year 2 69% 71% 71% 73% 73% 71% 70% 71%

% in AR Year 5 62% 63% 63% 64%

% in AR Year 1 72% 70% 70% 71% 72% 72% 70% 73%

% in AR Year 2 67% 66% 65% 66% 66% 67% 65% 69%

% in AR Year 5 61% 60% 57% 59%

% in AR Year 1 48% 51% 49% 52% 48% 63% 62% 62%

% in AR Year 2 44% 50% 48% 51% 62% 62% 61% 62%

% in AR Year 5 55% 59% 56% 61%

% in AR Year 1 53% 47% 59% 52% 55% 62% 49% 52%

% in AR Year 2 46% 40% 53% 43% 48% 54% 42% 45%

% in AR Year 5 45% 32% 44% 42%

New Version

Technical Certificate

Associate’s

Bachelor's

Master's

Doctoral-1st Professional

Doctoral-Research/

Scholarship

Past Graduates Recent Graduates

Graduation Year

Certificate of Proficiency

Average Salary vs. Median Wages

28

OLDAverage Wages

= Sum(4th Quarter Wages * 4)/# Employed

NEWMedian Wages Y1

= Median of Sum(Q1+Q2+Q3+Q4)

Original Report

Degree LevelAverage Salary Year 1

2006 2007 2008Certificate $ 19,396 $ 19,876 $ 18,639 Associate $ 18,868 $ 19,261 $ 19,244 Bachelor $ 26,860 $ 27,332 $ 26,116 Master $ 40,767 $ 41,274 $ 42,305 Doctoral $ 54,966 $ 54,707 $ 54,153 1st Professional $ 44,102 $ 40,551 $ 46,888

Median Wages Year 1Grad. Year 2006 2007 2008 2009Wage Year 2007 2008 2009 2010

CP $ 12,693 $ 13,370 $ 12,657 $ 11,203 TC $ 20,656 $ 20,396 $ 20,261 $ 20,140 AA $ 16,128 $ 16,468 $ 17,350 $ 16,519 BS $ 21,232 $ 21,574 $ 20,831 $ 19,708 MS $ 37,735 $ 38,593 $ 39,780 $ 39,583

1st Pro $ 35,998 $ 28,004 $ 44,250 $ 45,356 PhD $ 50,059 $ 48,981 $ 47,728 $ 44,511

Median Wages Year 2Grad. Year 2006 2007 2008 2009Wage Year 2008 2009 2010 2011

CP $ 16,459 $ 17,207 $ 15,272 $ 14,306 TC $ 23,943 $ 23,650 $ 23,206 $ 23,240 AA $ 19,747 $ 19,380 $ 19,949 $ 19,449 BS $ 28,162 $ 28,749 $ 27,595 $ 27,217 MS $ 41,104 $ 41,842 $ 42,928 $ 42,721

1st Pro $ 42,495 $ 48,347 $ 48,424 $ 49,392 PhD $ 53,900 $ 55,079 $ 54,430 $ 53,225

Median Wages Y2= Median of Sum(Q5+Q6+Q7+Q8)

Average Quarters Employed

29

OLDNot Used

NEWAverage Quarters Employed

=Sum(Quarters Employed)/# Employed

Average Quarters Employed Year 1 Average Quarters Employed Year 2Grad. Year 2006 2007 2008 2009 2006 2007 2008 2009Wage Year 2007 2008 2009 2010 2008 2009 2010 2011

CP 3.21 3.23 3.25 3.29 3.35 3.41 3.38 3.38TC 3.61 3.55 3.57 3.55 3.66 3.63 3.65 3.60AA 3.51 3.51 3.52 3.50 3.58 3.53 3.57 3.55BS 3.47 3.45 3.46 3.45 3.59 3.61 3.58 3.58MS 3.64 3.68 3.69 3.71 3.77 3.79 3.80 3.78

1st Pro 3.44 2.90 3.62 3.55 3.05 3.78 3.83 3.79PhD 3.51 3.65 3.47 3.52 3.76 3.65 3.68 3.76

Average Salary vs. Median Wages

30

OLDAverage Wages

= Sum(20th Quarter Wages * 4)/# Employed

NEWMedian Wages Y5

= Median of Sum(Q17+Q18+Q19+Q20)

Original Report

Degree LevelAverage Salary Year 5

2002 2003 2004Certificate $ 25,921 $ 26,008 $ 24,317 Associate $ 28,902 $ 29,837 $ 29,835 Bachelor $ 35,866 $ 36,766 $ 37,227 Master $ 45,277 $ 46,836 $ 47,888 Doctoral $ 63,028 $ 61,145 $ 66,059 1st Professional $ 78,095 $ 72,426 $ 79,324

Median Wages Year 1Grad. Year 2002 2003 2004 2005Wage Year 2003 2004 2005 2006

CP $ 14,995 $ 12,864 $ 11,909 $ 11,865 TC $ 16,397 $ 17,650 $ 18,392 $ 20,285 AA $ 14,509 $ 14,483 $ 15,120 $ 15,139 BS $ 17,186 $ 17,500 $ 19,325 $ 20,175 MS $ 32,417 $ 32,987 $ 34,398 $ 35,694

1st Pro $ 29,996 $ 35,647 $ 37,477 $ 31,500 PhD $ 44,673 $ 41,398 $ 45,897 $ 47,013

Median Wages Year 2Grad. Year 2002 2003 2004 2005Wage Year 2004 2005 2006 2007

CP $ 18,382 $ 17,044 $ 15,662 $ 15,790 TC $ 20,161 $ 20,786 $ 21,836 $ 23,023 AA $ 17,874 $ 17,122 $ 17,798 $ 18,089 BS $ 23,908 $ 24,923 $ 26,998 $ 27,404 MS $ 35,785 $ 37,496 $ 39,732 $ 39,228

1st Pro $ 43,731 $ 46,101 $ 48,638 $ 47,242 PhD $ 47,059 $ 48,477 $ 54,713 $ 55,730

Median Wages Year 5Grad. Year 2002 2003 2004 2005Wage Year 2007 2008 2009 2010

CP $ 26,504 $ 26,445 $ 21,794 $ 22,176 TC $ 26,920 $ 26,312 $ 27,865 $ 29,282 AA $ 29,899 $ 30,017 $ 30,617 $ 29,851 BS $ 36,580 $ 37,644 $ 37,762 $ 37,797 MS $ 46,555 $ 47,114 $ 48,648 $ 47,574

1st Pro $ 82,954 $ 83,162 $ 81,453 $ 68,626 PhD $ 59,657 $ 67,444 $ 67,665 $ 67,169

Average Quarters Employed

31

OLDNot Used

NEWAverage Quarters Employed

=Sum(Quarters Employed)/# Employed

Average Quarters Employed Year 1 Average Quarters Employed Year 2 Average Quarters Employed Year 5Grad. Year 2002 2003 2004 2005 2002 2003 2004 2005 2002 2003 2004 2005Wage Year 2003 2004 2005 2006 2004 2005 2006 2007 2007 2008 2009 2010

CP 3.33 3.20 3.09 3.09 3.50 3.36 3.35 3.37 3.60 3.62 3.52 3.50TC 3.56 3.60 3.60 3.59 3.67 3.62 3.67 3.64 3.71 3.68 3.71 3.68AA 3.52 3.50 3.51 3.52 3.58 3.56 3.56 3.53 3.72 3.71 3.73 3.71BS 3.42 3.40 3.45 3.47 3.55 3.56 3.60 3.59 3.73 3.75 3.75 3.74MS 3.67 3.66 3.65 3.66 3.76 3.77 3.75 3.73 3.81 3.82 3.84 3.82

1st Pro 3.25 3.44 3.50 3.32 3.68 3.61 3.69 3.69 3.69 3.76 3.76 3.65PhD 3.49 3.36 3.58 3.56 3.72 3.64 3.70 3.79 3.64 3.97 3.86 3.73

Wages by Degree Level by Major

32

OLDAverage Wages

= Sum(4th Quarter Wages * 4)/# Employed

NEWMedian Wages 2010

= Median of Sum2010(Q1+Q2+Q3+Q4)Original Report-HEALTH SCIENCES MAJORS

Degree LevelAverage Salary Year 5 Average Salary Year 1

2002 2003 2004 2006 2007 2008Bachelor $ 43,713 $ 44,882 $ 47,581 $ 37,448 $ 38,034 $ 38,317

Years Since Graduation 1 Year 2 Years 3 Years 4 Years 5 Years 6 Years 7 Years 8 Years

Graduation Year 2009 2008 2007 2006 2005 2004 2003 2002

HEALTH PROFESSIONS AND RELATED PROGRAMS

CP%Employed 81% 79% 79% 75% 75% 77% 71% 72%

Median Salary 11,662 15,007 16,612 22,374 21,432 19,449 22,608 28,253

TC%Employed 86% 88% 85% 82% 79% 79% 76% 70%

Median Salary 25,676 25,593 28,131 29,402 29,274 30,220 31,247 31,587

AA%Employed 83% 75% 76% 75% 73% 71% 69% 67%

Median Salary 38,144 39,159 40,314 42,691 41,982 43,568 44,713 45,641

BS%Employed 76% 70% 68% 68% 68% 66% 67% 63%

Median Salary 37,067 39,156 43,681 44,751 44,711 48,977 48,200 46,942

MS%Employed 66% 63% 60% 53% 55% 61% 58% 58%

Median Salary 55,070 52,754 58,686 58,532 55,550 55,503 58,551 56,490

1STPRO%Employed 62% 67% 64% 68% 64% 57% 58% 61%

Median Salary 56,939 52,351 80,197 105,787 111,052 118,980 138,125 126,844

PHD%Employed 69% 79% 75% 79% 65% 54% 38% 46%

Median Salary 56,000 57,927 62,482 60,479 66,671 68,803 75,749 78,659

Wages by Degree Level by Major

33

OLDAverage Wages

= Sum(4th Quarter Wages * 4)/# Employed

NEWMedian Wages 2010

= Median of Sum2010(Q1+Q2+Q3+Q4)

1 Year 2 Years 3 Years 4 Years 5 Years 6 Years 7 Years 8 Years

2009 2008 2007 2006 2005 2004 2003 2002

%Employed 79% 78% 90% 73% 80% 80%

Median Salary 10,446 13,761 14,222 17,634 17,522 18,577

%Employed 88% 91% 78% 76% 73% 80%

Median Salary 12,949 14,419 12,680 15,816 20,437 18,846

%Employed 78% 84% 81% 86% 79% 85% 79% 95%

Median Salary 10,908 14,737 20,737 21,795 23,027 23,456 29,082 24,039

%Employed 88% 85% 82% 83% 79% 79% 77% 70%

Median Salary 30,666 33,468 35,166 35,973 36,526 37,906 38,947 39,744

%Employed 87% 85% 80% 80% 78% 77% 75% 74%

Median Salary 41,374 43,624 44,555 45,295 46,750 49,226 48,834 49,690

%Employed 86% 83% 81% 78% 76% 83% 77% 68%

Median Salary 63,659 57,067 62,445 61,426 64,973 63,253 69,595 66,009

%Employed 53% 55% 63% 66% 56% 69% 44% 57%

Median Salary 58,331 72,495 68,985 72,351 71,961 79,259 74,227 73,275

BS

MS

1STPRO

EDUCATION

Years Since Graduation

Graduation Year

CP

TC

AA

PHD

Original Report-EDUCATION

Degree LevelAverage Salary Year 5 Average Salary Year 1

2002 2003 2004 2006 2007 2008Bachelor $34,791 $36,121 $36,218 $30,233 $30,806 $30,534

UAMS Study

Arkansas Research Center

Contact Information

arc.arkansas.gov

36