Post on 20-Jan-2016
description
ATTC Network Evaluation
1
National Evaluation of the Addiction Technology Transfer Center (ATTC) Network:
ATTC Event & Activity Reporting Database Review
Roy M. Gabriel, Ph.D.Jeffrey R.W. Knudsen Margaret Gwaltney, M.B.A. Richard Finkbiner, Ph.D.RMC Research Corporation Abt Associates, Inc. MANILA Consulting GroupPortland, OR Bethesda, MD McLean, VA
Presentation to ATTC Directors
November 5, 2009
ATTC NetworkEvaluation
2
Agenda
The ATTC Event & Activity Reporting Database: Review of past action steps
The NO, NET, and the ATTC Network Implementation to date Examination of current data
All events vs. GPRA only
Role of these data in overall evaluation Next steps???
ATTC NetworkEvaluation
3
Review—Where We’ve Been
ATTC NetworkEvaluation
4
ATTC Event & Activity Reporting Database and the NET
ATTC Event & Activity Reporting Database was augmented to meet multiple reporting needs: National Office/CSAT, NIDA, and the national evaluation.
In coordination with ATTC National Office, the NET developed a web-based query system to extract and summarize information from the database. Examples: Query the entire Network, specific region(s), time periods,
event types, etc. Tally number of events, number of participants, etc., as
ATTC “outputs” (both GPRA and non-GPRA). Cross-tab tallies w/many other event characteristics
(e.g., funding source, materials source, GPRA/non-GPRA, collaborator type, delivery mode, etc.).
ATTC NetworkEvaluation
5
The Underlying Premise . . .
During the design phase of the ATTC Evaluation contract, ATTC Directors clearly indicated that, from a service provision perspective, GPRA data did not tell the entire story.
It was clear that other very important services were being provided by ATTC centers, but were not being counted by the GPRA system.
ATTC NetworkEvaluation
6
The Output is Only as Good as the Input . . . Data Quality
The NET presented a series of sample summary tables at the December 2008 ATTC Directors meeting.
It was determined that assuring useful (and accurate) data queries and reports is contingent on the NET and the Network working together: The ATTC NO and the NET hosted a 2-part webinar series in
March 2009 reviewing (a) the purpose of the Event & Activity Reporting Database (from the perspectives of CSAT, NIDA, and the NET) and (b) data entry procedures.
The ATTC Performance Monitoring Subcommittee was working to (a) re-disseminate activity/event definitions and to (b) establish rules of thumb regarding what non-GPRA events should be entered into the system. The NO provided draft guidelines for review (April 2009).
ATTC NetworkEvaluation
7
NET Reporting Plans—On Hold
The NET prepared templates for quarterly summaries of events/activities to each regional ATTC. Reports were to feature regional activity data along
with a Network-wide summary of the same information.
The NET had hoped such summary reports would increase data utilization within Network.
Production and dissemination were contingent upon more consistent implementation of data system. The NET does not want to issue reports if numbers
are misleading, skewed, or simply inaccurate.
ATTC NetworkEvaluation
8
Implementation of Event & Activity Reporting Database
January 1–June 30, 2009
ATTC NetworkEvaluation
9
GPRA vs. Non-GPRA Events
GPRA? # of Events % of Events Valid % of Events
(Not Specified)a 29 4%
Yes 443 59% 62%
No 273 37% 38%
Totalb 745
aNote: This variable is only required once an event has occurred. “Not specified” responses likely represent event records not updated after the event took place.
bNote: Unduplicated event count.
ATTC NetworkEvaluation
Illustrative NET Interpretation
In a 6-month period, the Network conducted nearly 750 events and activities. An average of about 50 events/activities per region.
Much variability in these numbers from region to region.
Nearly 40% of these are not captured by the GPRA reporting system.
“Natural” follow-up questions: What kind of events?
Provided to whom? With whom?
On what topics?
10
ATTC NetworkEvaluation
11
Variability in Entering Non-GPRA Events
3 Regional Centers account for 70% of all non-GPRA events entered into database. These 3 Regional Centers demonstrated a 25%
GPRA/75% non-GPRA balance.
Time Period 0 events 1- 5 events 6- 10 events 10- 15 events 16 + events
Yr2 Q2 3 regions 8 regions 0 regions 0 regions 3 regions
Yr2 Q3 4 regions 4 regions 3 regions 1 regions 2 regions
Yr2 Q2 max/min (non-GPRA Events entered): 54/ 0 Yr2 Q3 max/min (non-GPRA Events entered): 30/ 0
ATTC NetworkEvaluation
NET Observations/Questions
Is it accurate that: 3–4 regions conducted no non-GPRA
events in these quarters? The majority of regions conduct fewer than
5 non-GPRA events in a quarter? In contrast to the earlier slide that indicated,
“Network-wide,” nearly 40% of the events conducted fall in the non-GPRA category.
What is the balance between GPRA and non-GPRA events in the Network: 60%/40%? 25%/75%? Something in between?
12
ATTC NetworkEvaluation
13
A Closer Look at Data:Event TypeYR2, Q2–3
ATTC NetworkEvaluation
14
Event Type (n = 745)
16%
0%
60%
1%
2%
4%
10%
7%
Meeting
Assessment
Educational Event
Formal Presentation
Conference
Brief Tech Asst
Tech Asst
Other
ATTC NetworkEvaluation
15
Event Type—GPRA Only (n = 443)
88%
5%3%1%
3%
Meeting
Assessment
Educational Event
Formal Presentation
Conference
Brief Tech Asst
Tech Asst
Other
ATTC NetworkEvaluation
16
Event Type—Non GPRA Only (n = 273)
15%
0%
35%
13%
24%
2%1%10%
Meeting
Assessment
Educational Event
Formal Presentation
Conference
Brief Tech Asst
Tech Asst
Other
ATTC NetworkEvaluation
17
Event Types for 3 Centers Consistently Logging Both GPRA & Non GPRA Events (n = 256)
32%
0%
22%
11%
23%
1%2%9%
Meeting
Assessment
Educational Event
Formal Presentation
Conference
Brief Tech Asst
Tech Asst
Other
ATTC NetworkEvaluation
18
A Closer Look at Data:Collaborator Type
YR2, Q2–3
ATTC NetworkEvaluation
19
Collaborator Type (n = 745 events)
6%
5%
35%
5%4%1%
14%
1%
4%
12%
13%
Another ATTC
CTN Node
SSA
Community Tx Programs
Provider Associations
Recovery Org
State/Regional Org
National Org (non-Fed)
Federal Org
Other
None
ATTC NetworkEvaluation
20
Collaborator Type—GPRA Only (n = 443)
6%
7%
25%
7%
4%1%16%
2%
5%
14%
13%
Another ATTC
CTN Node
SSA
Community Tx Programs
Provider Associations
Recovery Org
State/Regional Org
National Org (non-Fed)
Federal Org
Other
None
ATTC NetworkEvaluation
21
Collaborator Type—Non GPRA Only (n = 273)
5%3%
54%
2%
3%
0%
10%
0%
3%
6%
14%
Another ATTC
CTN Node
SSA
Community Tx Programs
Provider Associations
Recovery Org
State/Regional Org
National Org (non-Fed)
Federal Org
Other
None
ATTC NetworkEvaluation
22
Collaborator Types for 3 Centers Consistently Logging Both GPRA & Non GPRA Events (n = 256 events)
5%2%
45%
5%2%
0%
25%
1%
2%
3%
10%
Another ATTC
CTN Node
SSA
Community Tx Programs
Provider Associations
Recovery Org
State/Regional Org
National Org (non-Fed)
Federal Org
Other
None
ATTC NetworkEvaluation
23
Many More Examples Available . . .
Previous slides provided only as illustrative examples.
Many other queries could have also been used to demonstrate the impact of non-GPRA data, for example: Primary funding source Tech transfer objective Delivery mode Target audience State-level service saturation
ATTC NetworkEvaluation
24
What This Means . . .
The inclusion of non-GPRA data changes the overall service delivery “picture” for the Network. This is not a revelation, nor unexpected; but nonetheless,
from the NET’s perspective, very important to the Network.
The change in picture is even more apparent when looking at those few Centers who seem to be higher implementers (or reporters) of non-GPRA activities.
However, given the variability in reporting, the NET can only confidently put forth the service delivery picture drawn from GPRA. When 70% of non-GPRA entries are coming from 20% of the
Network, the climate for skew exists.
ATTC NetworkEvaluation
Importance of Event & Activity Reporting Database in National Evaluation
The NET has designed a comprehensive series of data collection activities to: Address and assess multiple objectives of tech
transfer in the Network.
Obtain a variety of perspectives on the effectiveness and value of the ATTC Network.
Across all of these data collection activities, the database is the only one that can capture the sheer volume of work produced by the Network (and volume is valued by many key stakeholders).
25
ATTC NetworkEvaluation
26
Next Steps
The NET will run all developed queries on GPRA events entered into ATTC Event & Activity Reporting Database to learn more about planning processes, partners, funding, materials, objectives, etc. of ATTC provided services.
If the NET can determine that entry of non-GPRA events/activities has improved, pool of events targeted for inquiry will be expanded.