Statewide AT Program Leaders SymposiumUsing Data and the Program Review Manual for Systematic Program Improvement

May 17, 2011

Sara Sack--Kansas

Using Data & The Review Manual for Program Evaluation

Kansas has a “thing” about data—validity and reliability are extremely important “Collecting good quality data plays a vital role in

supplying objective information for the problems under study so that some analytical understanding of the problems and hence solutions can be obtained.”

“Making decision on the basis of poor quality data is risky and may lead to disastrous results, as the situation may be distorted and hence all subsequent analyses and decision making will rest on a shaky ground.” (Hong Kong Polytechnic University)

Using Data & The Review Manual for Program Evaluation Data quality involves ensuring the

accuracy, timeliness, completeness and consistency of data used by an organization while also making sure that everyone who uses the data has a common understanding of what the data represents. (Andrew Greenyer)

Kansas ATK staff achieve 85% reliability on 75 item coding task before being allowed to enter data into database.

Analyzing AT Data: Management Weekly management level meeting with rotating

agenda Review grants & contract obligation & performance AT Access Site Data Review Collaborations & Partnerships Communication & Public Awareness

Electronic database (and entry required by the 5th of the month for all subcontractors) makes data pull easy

Admin level reports provide data on their AT Site’s performance: completed goals by AT category, age, county, and outcomes collection within a specified date range

Super user level has access to all reports 60:40 performance easy to obtain

ATK – 60:40 Split

Reuse (Request)34%

Demonstra-tion9%

Fund15%

Loan19%

Info & Assist10%

Training6%

Assessment8%

ATK Completed Goals (77%) Oct 2009 - Sept 2010

Big 4

Reuse (Request)

Demonstration

Fund

Loan

Other

Info & Assist

Training

Assessment

Southeast AT Access SiteParsons (SKIL)

Reuse (Request)30%

Demonstration

14%

Fund29%

Loan4%

Info & Assist12%

Training8%

Assessment4%

SE AT Site Completed Goals (77%) Oct 2009 - Sept 2010

Big 4

Reuse (Request)

Demonstration

Fund

Loan

Other

Info & Assist

Training

Assessment

Analyzing AT Data: All Staff

All ATK staff meet every other month generally face to face (webinars in the winter) to obtain training and discuss data and performance

So….you would think we would have program evaluation pretty well covered, right?

Utilizing the Review Manual for Program Evaluation

Review process requires the program to compare 3 years of data. ATK’s data review generally compared performance to the previous year or to the same quarter of the previous year

Review process also requires programs to review their financial data for a three year period.

Looking at a three year period provided additional, important information

ATK-KEE REUTILIZATION DONATIONS

2007-2008 2008-2009 2009-2010

Medicaid 32 $73,884

52 $64,786

70$128,839

Public/private

670 $622,243

852$866,665

716$851,388

KRS (low vision)

7$8,100

702 $696,127

904$931,451

793$988,327

REUTILIZATIONAT Category 2008

N = 6202009

N = 6782010

N = 698

Vision 2 – 3,000 1 – 2,100 8 – 12,480

Hearing 0 0 0

Speech Comm. 3 – 20,100 5 – 26,790 5 – 12,030

Learning, Cognition

3 – 3,600 3 – 2,508 0

Mobility, Seating

288 – 416,793 281 – 538,705 337 – 637,912

Daily Living 308 – 221,053 382 – 288,128 345 – 275,246

Environmental Adapts

0 1 – 3,200 0

Vehicle Mods. 0 0 0

Computers & Related

15 – 10,334 5 – 10,040 3 – 2,336

Recreation, Sports

1 – 1,000 0 0

TOTAL $675,880 $871,471 $940,004

What New Information Did We Gain From This Three Year Analysis?

In the area of Demonstration, we saw that our efforts to educate policymakers about the value of demonstration really were beginning to take hold and pay off

In Device Loan we were reminded of our efforts to “act super frugally” and invest savings into equipment purchases

AT Access Sites were reminded of the positive effects of specific Public Awareness Activities and vowed to try some of those efforts again

What New Information Did We Gain From This Three Year Analysis?

In Financial Loan of our AFP, the three year scan highlighted shifts in applications by equipment categories more clearly than just the two year scan

The funding support service that ATK provides as part of the State Financing showed major shifts in funders’ support for various technologies

What New Information Did We Gain From The Analysis of Three Years of Financial Data?

In the breakout and analysis of cost allocations trends could be identified and, as with other sections, staff recalled particularly strong efforts and activities.

In the area of Transition, the three year view of data showed how much was accomplished with very little money

Program Improvement Initiatives Based on Self-Evaluation & Program Review

Reviewers are most likely in the audience so we don’t want to tip our hand but…. Continue to refine our online database—more

automated features so staff have more time for consumers

Continue to analyze use of time—productivity is high so want to increase focus on “difficult to solve” situations

As our state moves increasingly to a “community based service model” we want to always be a part of the community solution

Program Improvement Initiatives Based on Self-Evaluation & Program Review

Conduct a detailed data analysis regarding our Public Awareness efforts and figure out if they are advancing our community based service efforts and our efforts to reach consumers that are in areas that receive fewer services

Work with the data to see if we can determine if consumers are utilizing a continuum of services

Questions?

Contact: Sara Sackssack@ku.edu