Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health...
Transcript of Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health...
![Page 1: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/1.jpg)
Current Examples of Joint HSR/SR Evaluation
Edward J. Miech, Ed.D.Research Scientist
Health Services Research & DevelopmentVA-CASE (VISN11 VERC)
Roudebush VA Medical CenterIndianapolis, IN
July 14, 2010
![Page 2: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/2.jpg)
VA Cancer Care Collaborative, Phase II
• 21 VA Teams Working on 5 Different Cancer Types– Lung, Prostate, HCC, Head & Neck, Colon
• SR-based Collaborative• Each team has a coach who is experienced with
SR, team dynamics, VA context
![Page 3: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/3.jpg)
VA Cancer Care Collaborative, Phase II
• Each site receives monthly visits by an Industrial Engineer
• Learning Sessions include sessions on SR principles, concepts & tools
• Successor to VA Cancer Care Collaborative-Phase I
• Co-Chairs: Ken Klotz and Kim Voss
![Page 4: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/4.jpg)
Team Development Measure
• CCC II planning committee interested in learning more about team dynamics
• In May 2010 (during Action Period 1) individual CCC II team members were invited to complete online version of Team Development Measure
• n=97; 18 Teams ≥ 3 members completed TDM
![Page 5: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/5.jpg)
Team Development Measure
• 31-item instrument where respondents rate level of agreement with statements about team cohesion, communication, roles and goals
• Respondents retain anonymity• Takes 5-10 Minutes To Complete
![Page 6: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/6.jpg)
Team Development Measure
• Examples of Statements– Team members say what they really mean. – The goals of the team are clearly understood by
all team members. – The work I do on this team is valued by the other
team members.
• 4-point Likert response scale: – Disagree Strongly, Disagree, Agree, Agree Strongly
![Page 7: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/7.jpg)
Team Development Measure
• Permission Granted To Use Freely In VA• VA Guide To Using TDM Developed Under
Leadership of DeDe Ordin and Brian Mittman – 12-page detailed handbook, includes questions– published February 2010
• Freely Available on QUERI website at http://www.queri.research.va.gov/ciprs/TDM_Module.pdf
![Page 8: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/8.jpg)
Team Development Measure
Preliminary Results
![Page 9: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/9.jpg)
![Page 10: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/10.jpg)
![Page 11: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/11.jpg)
Team Development Measure
• Coaches Receive Customized Reports For Each of 18 CCC II Teams– Team Reports will include distribution of
individual responses for each TDM item– Visual depiction of members’ different
perceptions of current state of team
![Page 12: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/12.jpg)
Team Development Measure
Example of TDM Customized Team Report
![Page 13: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/13.jpg)
Number of respondents = 6Mean score = 65Lowest score = 50Highest score = 88 The graph below shows how many team members see the team at what stage.
Baseline, June 2010
![Page 14: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/14.jpg)
INSERT SNAG IT OF TDM HERE
![Page 15: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/15.jpg)
INSERT SNAG IT OF TDM HERE
![Page 16: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/16.jpg)
INSERT SNAG IT OF TDM HERE
![Page 17: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/17.jpg)
Team Development Measure
• Coaches have option to lead 1-hour facilitated discussion
• Facilitated discussion led by coach– Where team is now– What next stage looks like– How the team might get to next stage
![Page 18: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/18.jpg)
Team Development Measure
• Data captured for TDM is quickly analyzed and reported back to CCC planning committee
• TDM Data Cross-Referenced With Other Measures
• At level of team, 1. diverse perceptions of how members see the
team leads to 2. an evidence-based discussion that3. promotes further team development and growth
![Page 19: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/19.jpg)
Team Development Measure
• Tool = Online Instrument (Kwik Surveys)• All-Digital Data Capture and Organization– Downloads automatically into Excel
• Respondents Retain Individual Anonymity – Allows for candid responses to questions
• Multi-Level Data– Tracked at both individual and team level– Summary Reports Useful at both Collaborative
and Team levels
![Page 20: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/20.jpg)
PCMH Collaborative LS1 - West
• Learning Session with approximately 300 participants taking place right now in San Diego
• 300-clicker ARS system in use to capture baseline and endpoint data (i.e., pre- and post-)
![Page 21: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/21.jpg)
Baseline ARS at PCMH LS1 - West
Selected Items
![Page 22: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/22.jpg)
How many collaboratives have you participated in, including this one?
1 2 3 4 5 6
41%
36%
10%
4%3%6%
1. 12. 23. 34. 45. 56. 6 or more
![Page 23: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/23.jpg)
How many years have you worked for the VA?
1 2 3 4 5 6 7
7%
16%18%
8%
16%
14%
21%
1. < 12. 1 - 23. 3 - 54. 6 - 105. 11 - 156. 16 - 257. 26 +
![Page 24: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/24.jpg)
What is your job title?
1 2 3 4 5 6 7 8 9 10
18%
1%
8%
26%
9%
2%3%
18%
6%
8%
1. Administrative Assoc2. HPDP/BHC3. LVN/LPN4. Nurse5. PCMH Faculty & Staff6. Pharmacist7. Physician8. Social Worker9. Technician/MA10. VA Faculty Leader
![Page 25: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/25.jpg)
How many VA project-based teams (QI, Systems Redesign, Lean, etc.) have you personally been a
member of, including this one?
1 2 3 4 5 6
26%
20%
29%
3%
8%
15%
1. 12. 23. 34. 45. 56. 6 +
![Page 26: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/26.jpg)
Did you complete the prework before you came to LS #1?
1 2 3 4 5 6
54%
15%
4%2%
10%
16%
1. Yes, all2. Yes, most3. Yes, some4. No5. Did not know about
pre-work6. N/A
![Page 27: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/27.jpg)
I had enough knowledge about the tools available to complete the prework.
1 2 3 4 5
12%
34%
5%
18%
31%1. Strongly Agree2. Agree3. Neutral4. Disagree5. Strongly Disagree
![Page 28: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/28.jpg)
I had enough resources, including time, to complete the prework.
1 2 3 4 5
7%
20%23%
41%
10%
1. Strongly Agree2. Agree3. Neutral4. Disagree5. Strongly Disagree
![Page 29: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/29.jpg)
![Page 30: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/30.jpg)
![Page 31: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/31.jpg)
Exploratory Data Analysis: Correlations
• “How many Collaboratives have you participated in?” and– How many project-based teams have you been
on? (r = .55, p<.0001)– Did you complete the LS1 pre-work? (r = .21
p<.001)
![Page 32: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/32.jpg)
![Page 33: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/33.jpg)
ARS data from CCC II LS1-New Orleans
• Statistically significant correlations between participation in any prior VA Collaborative and– Higher levels of self-reported familiarity with
Systems Redesign tools (r=.59, p>.0001)– Higher levels of perceived support from site
leadership for Collaborative work (r=.42, p<.02)– Higher letter grades given by participants to LS1
(r=.41, p<.02)
![Page 34: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/34.jpg)
Common Features: ARS & Online TDM
• All-Digital Capture (with built-in reporting features)
• Respondents retain anonymity• Responses can be tracked at individual level• Data can be segmented, aggregated, analyzed
for correlations, etc.
![Page 35: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/35.jpg)
![Page 36: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/36.jpg)
VA Yellow Belt Training Program
• In VA Yellow Belt training program participants actively learn about Systems Redesign and Lean principles, concepts, and tools through simulations, group application exercises, didactic presentations, and discussion
• Enrollment capped at 50 participants per session • Courses usually taught by faculty from the
Purdue PharmaTAP program • Sessions generally take place for 3.5 consecutive
days in an off-site location such as a hotel
![Page 37: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/37.jpg)
VA Yellow Belt Training Program
• Over 400 participants in VA Systems Redesign Yellow Belt training across 12 different sessions held between August 2009 (pilot in Indianapolis) and June 2010 (Boston, MA)
• Pre- and post-data collected at 11 of these 12 sessions
![Page 38: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/38.jpg)
Example: Interim YB Findings
![Page 39: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/39.jpg)
Example: Interim YB Findings
• Analyses found extremely strong correlations between:– the items “I believe this Yellow Belt workshop will
be relevant to my job” and “I am positive about Systems Redesign” (r=.68, p<.0001);
– the items “I found this workshop to be practical” and the overall letter grade given to the workshop (r=.67, p<.0001)
![Page 40: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/40.jpg)
Example: Yellow Belt in Connecticut
• I am familiar with many of the tools used in Systems Redesign. (n=34)
![Page 41: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/41.jpg)
ARS Advanced Features
• Multi-Voting• Priority Ranking – 1st choice =10 points– 2nd choice =5 points– 3rd choice= 1 point
• Impact-Effort Matrix– Graph two dimensions along x-axis and y-axis
• Conditional Branching
![Page 42: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/42.jpg)
Next Frontier
• Capturing Multi-Level Data From Complex, Multi-Site Initiatives in a Single Project File
• Flexibly Interrogating/Analyzing Evidence– Keywords– Codes– Indexing
![Page 43: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/43.jpg)
Next Frontier
• Adding Value to Source Data with – Applying codes and tags– Linking to Analytic Memos– Performing Context Matrix Analyses
• Rapid Reporting of findings/results to initiative organizers/planners on rolling basis
• Capturing, Organizing, Analyzing and Reporting Rich-Context Data occurs on ongoing, rolling basis
• Digital Tool = NVivo8
![Page 44: Current Examples of Joint HSR/SR Evaluation Edward J. Miech, Ed.D. Research Scientist Health Services Research & Development VA-CASE (VISN11 VERC) Roudebush.](https://reader035.fdocuments.us/reader035/viewer/2022062713/56649ce55503460f949b3483/html5/thumbnails/44.jpg)
Acknowledgements
Heather Woodward-Hagg– Systems Redesign, Chief, Roudebush VA– Co-Director, VA-CASE (VISN11 VERC)
Mary Sherrill– Co-Director, VA-CASE (VISN11 VERC)
Dawn Bravata– Co-Coordinator, Stroke QUERI