Simon de Lusignan Reader in General Practice & Informatics [email protected] General Practice &...

Simon de Lusignan

Reader in General Practice & Informatics

[email protected] General Practice & Primary CareDivision of Population Health Sciences & EducationSt George’s – University of London

Pushpa Kumarapeli

Lecturer in Computing and Information [email protected] of Computing, Information Systems and Mathematics,Kingston University, London

Head-to-head comparison Head-to-head comparison ....using ALFA....using ALFA

ALFA (Aggregation of Log Files for Analysis)An open source toolkit for the precise observation of the clinical consultation & the impact of using technology

www.biomedicalinformatics.info/alfa/www.clininf.eu

Pushpa’s academic work

PhD research on – process modelling of computerised clinical consultations; a video study– Developing the ALFA method as a tool-kit for consultation observation, analysis and evaluation – Design and development of the software tools; to record time structured observational data (the ODC tool), recording of doctor’s computer use

(UAR), verbal interactions (VAR), and designing of a tool to aggregate observations from multiple sources (LFA)– Developing a UML (Unified Modelling Language) specification and XML schema to produce outputs useful for software engineers/system

designers

Worked with Simon on various research projects that were based on ‘routinely collected general practice data’– Data extraction using MIQUEST, query authoring, developed a system for auto generation of meta-data, experienced in cleaning, processing and

analysis of data from GP systems

Lecturer in Computing and Information Systems– Database management systems, software quality, system analysis and design

(a former dental student, qualified as an Information System Designer, worked at Sun Microsystems as a Webtechnologist)

My practice & academic work

GP in Guildford (30 miles SW of London)– 11,800 patient practice– 6.5 Whole time equivalent GPs / 8 partners– Computerised since 1988 – EMIS brand since 1994– Involved in “Practice Based Commissioning” and UK’s

First ICO (integrating Care Organisation)

Head of GP & Primary Care, St. Georges, London– Teaching network - >200 practices >1,000 placements

– Primary Care Informatics (PCI) How IT enables quality improvement?

(1)Impact of IT on the consultation - ALFA (2) Using routinely collected data for QI

- Data linkage & pseudonymisation- Chronic Kidney Disease (CKD)

(3) Scholarship: - Chair PCI Working group of European Federation for Medical Informatics (EFMI) & UK rep to EFMI- Editor of Informatics in Primary Care

Just accepted Chair at University of Surrey– http://gp2.sgul.ac.uk

www.woodbridgehillsurgery.co.uk

1. Obsession with interoperability (or linkage) over usability

The great National Pilot of IT...

2. Time to put some science into developing usable systems...

Changes driven by better patient care

3. Observing and analysing the influence of computer on the clinical consultation

to support better information system solutions

Context

www.biomedicalinformatics.info/alfa/ www.clininf.eu

Interoperability (and linkage) ahead of usability...

England:– The National Programme for IT has achieved some successes

– Lot of emphasis on linkage / interoperability– Systematising development (Waterfall) rather than pragmatic with users (agile)

Scotland – Focus on record linkage (unique ID) – functional links between records

Loss of focus on:– System usability– Improving outcomes for patients – Usability within the consultation

1. Obsession with interoperability over usability

The great National Pilot of IT...

2. Time to put some science into developing usable systems...

Changes driven by better patient care

Context

1. Observing the consultation – the ALFA tool-kit

2. Understanding the computerised GP consultation – Overview of the consultation – content and context– time spent on performing core tasks– their associated process

3. Changing our approach to developing systems

Overview


Outline of the presentation

Part 1: Development of multi-channel video - ALFA toolkit– Developed video methods of observing the consultation– Additional tools developed:

– Keyboard & mouse use (User Action Recording)

– Time stamp to record start & end of speech (Voice Activation Recording)

– Coding tool (Observational data capture)

– Combining & managing may files (Log File Aggregation) Part 2: The clinical consultation is a messy business

– Initial results from -167 consultations -15 GPs – 4 EPR systems Part 3: The computer as the third actor

– The computer can initiate the consultation– How the stage is set is critical... ...if the computer is to play its part

Discussion– Time to inject some science into developing more usable systems– Serious about using the computer –change your consulting room layout...


Free on-line access to ALFAwww.biomedicalinformatics.info/alfa/

Supporting information

– Comparison with other tools– Overview of technical process– Demonstrations

– Manuals– Presentations– Publications

ALFA components free to download– ODC (Observational Data Capture)– VAR (Voice Activity Recorder)– LFA (Log File Aggregation tool)– UAR (User Activity Recorder)

Open source GPL licence– Freely available

1. Developing video methods for observing the consultation2. Considerable variation in GP clinical systems3. ALFA is needed because we lack an effective method to:

– Compare or develop EPR – Capture context of data recording

Part 1: Introduction & Rationale for ALFA


10-year development pathway of ALFA

1999 Video evaluation of new software – single video camera...

- Not enough data

2002 Multi-channel professional video

- Great but too expensive & cumbersome

Pattern recognition software

- Fun – but not yet able to make sense of results

2005 Cheap digital video

- Research resumes

2006 Qualitative study to explore number of channels & layout

Use of multi channel video to look at simulated consultations

- Lab environment (clinical skills rooms) felt wrong

- Computer use & room layout variable in own premises

- Developed utilities to allow additional data capture

2008 Created ALFA as an open source toolkit...

- Set up in 20 to 30 mins

- Data manageable aggregated onto just two files


2006 Low-cost digital video! No of Cameras & layout

Low cost video equipment– 4 channel video for 1000 Euros– Synchronisation “easy” No studio needed– High quality sound recording– Mix in free or low cost applications– Smaller

Qualitative study – How many cameras? What layout?– 3, 4 and 5 channels experimented with– Elite survey of medical educationalist used to assessing video– Need to see the patient’s body language

Leong A, Koczan P, de Lusignan S, Sheeler I. A framework for comparing video methods used to assess the clinical consultation: a qualitative study. Med Inform Internet Med. 2006 Dec;31(4):255-65.

Sheeler I, Koczan P, Wallage W, de Lusignan S. Low-cost three-channel video for assessment of the clinical consultation. Inform Prim Care. 2007;15(1):25-31.


2007/8 Simulated consultation – Lab & own consulting room

Compared three brands of EPR system, in simulated simple consultation - Scenario: A medication review and BP check

- Used a clinical skills lab- Trained students as raters - 4 to 5 raters assessed each consultation

- Computer used for >25% of the consultation, very high intra-class correlation coefficient between raters (>0.928, p<0.001)

- Limitations: “Contrived” environment; short duration items hard to code

Compared three brands of EPR system, in simulated consultation to assess CVS risk - Scenario: Two patients (actor) patients who needed their CVS assessing

- Some GPs systematic (form led) others personalised concentrating on one risk factor- Systematic: Mean 3.5 items coded taking 33.9% of consultation- Personalised: Mean 1.9 items coded taking 25.8% of consultation

- One GP used risk calculator on the Internet

Moulene MV, de Lusignan S, Freeman G, van Vlymen J, Sheeler I, Singleton A, Kumarapeli P. Assessing the impact of recording quality target data on the GP consultation using multi-channel video. Stud Health Technol Inform. 2007;129(Pt 2):1132-6.


2006-8 Three tools to capture detailed information & an output format for software engineers :

ODC – Observational Data Capture

– Allows coding of the multichannel video– Records room layout

UAR – User Action Recording– Captures the precise time each key is pressed & mouse

co-ordinates – Measures navigation/data entry ratios– Manual capture of precise times was challenging

VAR – Voice Activation Recording– Time stamps the start & end of speech. – Needed if linking to transcript text

Output a software engineer can interpret– UML sequence diagrams & XML data files

www.biomedicalinformatics.info/alfa/

Observational Data Capture (ODC)

Collection and analysis of observational data about doctor-patient and doctor-computer interactions

Watching the multi-channel video and identifying the occurrence of different interactions and their duration. e.g. Dr talking to Pt, Dr looking at the screen, Making eye contact, Prescribing

Method 1;•Watch the full video.•Set of key board keys assigned as a label for a specific interaction.•Appropriate key pressed to mark the start and end time of interactions

Method 2;•Go directly to the known part of the video.•Identify the start of the interaction, mark it,•Identify the end of interaction, mark it•Select the type of interaction from the drop down list


16

User Action Recording (UAR)

Recording clinician’s use of computer keyboard and mouse during the consultation

Captures the key strokes and the mouse movements

Value of the pressed key and the coordinates of the mouse pointer are written into two separate log files with time stamps

Time stamp

Key strokes

Location of mouse pointer (X,Y)


Voice Activity Recording (VAR) for linking to consultation transcripts

Time log of verbal interactions. • Log file created based on the doctor’s

and patient’s voice activity• Combining this with the transcription

gives a time stamped conversation log.

VAR interface and log file output

Consultation transcript

Time stamped consultation transcript


Kumarapeli P, de Lusignan S, Koczan P, Jones B, Sheeler I. The feasibility of using UML to compare the impact of different brands of computer system on the clinical consultation. Inform Prim Care. 2007;15(4):245-53.

Kumarapeli P, De Lusignan S, Ellis T, Jones B. Using Unified Modelling Language (UML) as a process-modelling technique for clinical-research process improvement. Med Inform Internet Med. 2007 Mar;32(1):51-64.

Output a software engineer could interpret!

UML: Unified Modeling Language – Sequence diagram

To develop these we also needed precisely time-stamped information about keyboard and mouse use....

Aggregating and analysing the data


Multiple files for analysis

One or more of the following files for analysis

UAR x2

– User Action Recording – Key board & Mouse

VAR– Voice Activation

Recording ODC

– Observational Data Capture


LFA Tool – links all results into two files:

File 1:

Multi-channel video recording (MCV)

User action recording (UAR)Up to 13 time stamped outputs

File 2:

Observational Data Capture (ODC)Voice Action Recorder (VAR)

Outputs:

Graphs and tables linked to coded events – linked to video files

Data for statistical packages

Data for software engineers

• Set up in <20 minutes

Setting up the ALFA toolkit

Hardware

Screen recording device(VGA2USB frame grabber)

Hard drive based camcorders(JVC Everio HDD 30GB)

Laptop for screen recording

Low profile tripods

ALFA kit packs into a rucksack and tripod bag

...readily deployable


Setting-up three cameras

High capacity batteries run for 3 hours

Optional additional web-cam

Use camera microphone

Adjust the three camera angels;1 - Dr’s upper body 2 - Patient’s upper body3 - Wide angle view of Dr, Patient and computer screen


Audio-Visual recording of consultation

Three video cameras record the consultation. Screen capture software records the clinical computer system’s monitor

output

1

24

3

Multi-channel video recording (MCV)


MCV Cameras Angles

Wide angle view

The camera capturing the wide angle shot should have a clear view of the Dr’s computer screenThis is important for synchronising the cameras footages with the screen recording

Patient’s camera

It is important to capture patient's hands

Doctor’s camera

Dr’s view should cover Dr’s upper body and the computer keyboard


Frame grabber copies monitor activity to a laptop

Locate the VGA graphics output of the Dr’s computer and disconnect it.

Connect the VGA splitter cable into the computers VGA output.

A laptop – usually under the Drs desk captures what happens on their computer


Producing the multi-channel video

• The three camera video channels are synchronised using a standard package

• Adobe Premiere Elements

Variation in brands of computer system


Variability between brands of EPR

UK brands of EPR appear to have developed organically

Four brands account for around 75% of the market– EMIS LV (Most used – old CHUI interface)– EMIS PCS (GUI version – auto-coding – 3 windows )– INPS Vision (4 windows, tabs and icons)– iSoft Synergy (3 windows, multi-tabs & icons )

If only we could combine the best features of each?


31

Variation in GP computer systems

EMIS LV EMIS PCS

INPS Vision iSoft Synergy


Variation in how BP is recorded in GP computer systems

EMIS LV EMIS PCS

INPS Vision iSoft Synergy


Variation in how cardiovascular risk is recorded:

EMIS LV EMIS PCS

INPS Vision

iSoft Synergy


Variation in coding in the consultation

Coding took place in >80%

No data were coded in:– EMIS LV (30%) – EMIS PCS (18%)– iSoft synergy (4.5%)– INPS Vision (7.4%)

In EMIS LV and PCS coding was less frequent– 1 item was coded in 69%– 2 items in 23%– Only 6 consultation 4 or more items

More items were coded in iSoft Synergy and INPS Vision– Mean 2.9 (iSoft) & 2.6 (vision) items per

consult– Mean 1.5 (LV) & 1.6 (PCS)

≤4 or ≥5 codes per consult

% consultations with no coded data


Contrasting methods used for clinical coding

in PCS - copying and pasting from previously entered codes from sub window for record summary

in Vision – selecting from main menu add item, needs more hand-eye coordination/interactions. Despite having number of icons for common coded data entry, they were rarely used


Contrasting time taken for clinical coding

Non-parametric test suggest that EMIS PCS is significantly faster than EMIS LV, or Vison (p<0.001); and no different than Synergy (p=0.083)

However, data linkage means more coding happens in Vision INPS and iSoft practices UML diagrams provide more insight...


1. Dr: Transition: Default Consultation ModeData entry

2. Dr: Select Heading: Problem: Double click3. EPR: Present Code list4. Dr: Navigation (Mouse): SubWin 1 SubWin2:

Problem List5. Dr: Selection: Problem List (Past History item)

Double click6. EPR: Add code { Problem List (Past History item)}7. Dr: Transition: Data Entry Coded Data Entry Free

Text

• Coded data entry embedded to free text data entry

• Less Transition time between Free text entry Coded entry Free text entry

• ‘Semantic Auto Completion’ feature: offers a picking list of coded data items as you type on


Vision1. Dr: Transition: Default Consultation ModeData entry2. Dr: Navigation (Mouse):SubWin 3: Coded entry 3. Dr:Type Search string4. Dr:Submit Search String5. EPR: Present item:16. Dr:Expand data entry7. EPR: Present data entry form8. EPR: Present picking list9. Dr: Select item10. Dr: Submit Item selection11. EPR: Accept Selection12. Dr: Transition: Coded Data entryFree text entry

• A small sub-window for Coded data entry displayed in default screen

• May need to expand the data entry window, if the desired coded data item is not presented as the first result in the picking list

• Initial transition time less, but may result in spending more time in code selection if the full picking list is needed

• Free text entry always linked to a coded entry

SynergyVision

LV

Variation in BP recording time

PCS – BP entered using other data entry forms as well Vision – standard BP entry form used always PCS – single method of activation used for the standard BP form Vision – different activation methods used, mostly determined by the current location

within the patient record


Variation in BP recording time Recording BP data in the EPR took between 5.9 and 27.6

seconds– xU 12.1 sec (x̃ 10.20) – SD 5 sec

PCS xU =13.6 (IQR=6.4), Vision xU =11.2 (IQR=1.5) PCS – GPs spend time reviewing, commented on past values Vision – default values left unchanged to minimise any delay


Consultation mode

Txt: BP

SW, NP, Code search

GP GPS:LV Patient

Exm: BP, Pulse

Select link: Exm

39sComments: SymptomsQ: Smoking status

A: Smoking status

Comment: BP values

SW, SP, Heading: Exm.

XSelect link: Code search

Txt: BPSW, NW, BP template

Ssy, Dias txt area

txt: Sys & Dias Comment: Symptoms

Comment: BP values, normal range

GP GPS:PCS Patient

Examination Request: BPExm Response: Agrees

Enter Txt: BPDisplay template: BPExamination: BP

Save BP results

Past BP, Normal range

Select Heading: Exm

Select, txt: Dias BPSelect, txt: Sys BP

Insert Exm header to EMR: BP

GP GPS:Vision Patient

Request: Examination findingsHgt, Wgt, BPSelect link BP

Past BP readings x 6Exm Request: BP

Exm Response: AgreesSelect: Add BPBP template

Date, Time, HCP, Read code, Position, Cuff detail, location

Examination: BP

Select, txt: Dias BPSelect, txt: Sys BP

Select, Code: BP normalComment: BP range

Select link: Past BP Save Exm results

Past BP readings x 6 Explains: BP change

Comment: BP values

GP GPS:Synergy Patient

Exm: BP, PulseQOF reminders

BP, Smoking, Smear, Summary

Comment: BP review

63s

Comment: Bp medicationSelect link: Add BP

BP template Free txt area

txt: Sys & Dias BPExm: Pulsetxt: Pulse, Ftxt

12s

UML diagrams allow software engineers to understand the process…


Acute prescribing time

55% (91/167) consultation have an acute precipitin

Little variation between brands – range 58% to 45%

The EMIS LV character user interface appears on video to offer advantages in speed of Rx

The mouse driven systems need more hand-eye coordination and were slowest where a completely new Px was started

PCS users (xU =26.3) spent slightly longer durations compared to Vision (xU =23.9)- Vision provided more default values for data entry- Vision also presented more prompts, mostly ignored


Encounter reviewing Encounter reviewing observed in 93% of consultations PCS has a sub window in the default consultation view, Vision users need

to navigate into a separate section Similar durations spent , (PCS; x̃ =30.6s, Vision x̃ =33.9s), but more

episodes noticed in Vision (x̃ =9) than in PCS (x̃ =5)


Free text entry & prompts Majority of free text entries in

Vision were associated with coded data entry – ‘strict problem orientation’

More patient history linked prompts in PCS.

Vision prompts & warnings mostly linked with prescriptions

Discussion


Summary

ALFA captures the context Key differences in time and process:

– Review of notes– Problem recording– Free text entry– BP recording– Acute prescribing

If only we could have a new system with all the best features....

ALFA should enable the development of more effective EPR systems to use in the clinical consultation.....


Reminder – change your consulting room layout!

The room layout inhibit the role of the computer as an actor

– Implications for shared decision making Consider inclusive layout with two seats

– 2/3rds time patient sits in semi-inclusive Dr chair– Younger patients more often sit in the inclusive chair


The evolution of the clinical consultation?

Doctor and Patient

Doctor , Patient ( and computer )Doctor, ( Patient ) and computer !

Handout:

Thanks for listening:Simon de Lusignan

[email protected]

Pushpa [email protected]


www.clininfo.eu

Simon de Lusignan Reader in General Practice & Informatics [email protected] General Practice &...

Documents

Transcript of Simon de Lusignan Reader in General Practice & Informatics [email protected] General Practice &...