Enhancing the learning opportunities of part-time postgraduate students whilst off-campus – periodic self-assessment of image

reporting skills using Blackboard.

Julie Nightingale and Ben MolloECE Conference 2007

Overview

Learning technologies project undertaken at the University of Salford in 2006-7.

• Overview of the context of the study (Medical Imaging)

• Outline of the educational programme

• Project design to address perceived programme limitations

• Pilot evaluation and delivery of the learning package

• Discussion of key findings and next steps

A traditional view of Medical Imaging

Radiographers : • registered health care professionals • responsible for production of high quality

medical images such as x-rays.

Radiologists : • registered specialist doctor• interprets the images and issues a

diagnostic report to the referring doctor

Catalysts for Change

In the last decade, a number of drivers effected changes in the traditional medical imaging workforce.

• increasing radiology workload

• shortage of UK radiologists

• enhanced educational status of radiographers

• changes in legislation

A contemporary view of Medical Imaging

Radiographers : • responsible for production of high quality

medical images such as x-rays. • increasingly involved in interpreting images

Radiologists : • interprets images and issues a diagnostic report to the referring doctor• more involvement in complex procedures including treatment

Established fields of radiographer interpretation

Ultrasound / Mammography / Skeletal trauma x-rays

Gastrointestinal imaging (colon and rectum)

Gastrointestinal imaging for suspected cancer

• Radiology examinations of the lower gastrointestinal tract are often undertaken in patients with suspected bowel cancer.

• 2nd most common cancer in the UK, and a major cause of cancer deaths (Cancer Research UK, 2004)

• Excellent prognosis -if identified and treated early

• Bowel cancer screening programmes introduced in several countries

Potential Problems in Diagnosis

• These radiology examinations are difficult to interpret due to a high level of perception errors (Markus et al,1990)

• Perception error – a false negative - the pathology is present but is not identified by the interpreter

• “Double reporting” recommended by the professional bodies – “two pairs of eyes are better than one!”

• Radiographers can be trained to contribute to this double reporting system.

Programme Overview

Postgraduate Certificate in Advanced Practice (Gastrointestinal Imaging)

• Commenced 2001 at University of Salford• First course in the UK to offer combined clinical and

academic training for the radiographer reporting role• Attracts students from across the UK and Ireland.

• 12 month block release programme• 1 week of university attendance every 3 months• Intervening period of clinical practice (radiologist mentor)• Students supported by email and Blackboard VLE

Previous Evaluations

• During lengthy clinical blocks students reported feelings of isolation and loss of motivation

• Noted variable clinical experience and lack of support from some mentors

• Some students became “entrenched” within the ways of working at their base hospital and the particular reporting style of their mentor.

• Potential for parochial inward-looking practitioner.

Project Outline

• Internally funded project (TLQIS)

• Blackboard VLE - previously used as a repository for information, but had potential to offer learning packages to encourage more active student engagement.

• Developed a Blackboard VLE learning package to introduce formative image interpretation and report writing opportunities

• Through this package, regular self, peer and tutor assessment of report writing accuracy and style was encouraged.

Phases of the Project

Case Acquisition

Case Verification

Developing the Teaching Package

Piloting

Delivery and External Review

Case Acquisition

• Acquired a bank of 30 suitable anonymised gastrointestinal cases (10 for pilot)

• Each case has up to 16 different images within it

• Each case has accompanying

clinical indications and diagnostic report

• Each case can be several megabytes in memory

Case verification

• Images in each case reviewed by a panel of “experts”

• Accompanying case report scrutinised and compared to the expert opinions, to ensure there was no ambiguity

• Where consensus could not be reached, alternative cases were sought

• This verification method follows traditional, quality assured practice within radiology

Developing the Teaching Package - case study database

Developing the Teaching Package – selection of a case

Developing the Teaching Package – case display

Piloting – campus-based training session

• Campus-based training session delivered by project co-ordinator • Purpose and layout of online package was explained.

• 8 students were guided through the first of 4 cases, then they wrote their findings (report) onto the relevant discussion board.

• Following completion of the 4 cases, the definitive (actual) report was placed on the discussion board.

• A facilitator re-visited each case with the students, encouraging exploration of their different reporting styles and highlighting examples of good practice.

Piloting – campus-based training session

• Campus-based training session delivered by project co-ordinator • Purpose and layout of online package was explained.

• Students were guided through the first of 4 cases, then placed their report on the relevant discussion board.

• Following completion of the 4 cases, the definitive (actual) report was placed on the discussion board.

• A facilitator re-visited each case with the students, encouraging exploration of their different reporting styles and highlighting examples of good practice.

Example of facilitator feedback

Campus based training session - evaluation by observation

Project co-ordinator observation study / facilitator feedback

• Poor viewing conditions (insufficient black-out curtains) interfered with image viewing.

• Poor image quality noted by one student was easily rectified by altering brightness and contrast settings on the computer monitor.

• Repetitive errors causing some frustration.

• Students needed no further assistance after 2 cases. • Mean of 13 minutes per case required. • All students appeared to benefit from the interactions.

Campus based training session - evaluation by questionnaire

A bespoke questionnaire administered to students- based upon some of the key principles of learning technologies and software evaluation (Neilson, 2005).

- piloted on two clinical colleagues who had recently been introduced to the training package.

Administered in 3 parts at beginning, middle and end of the session –

– Section 1 (demographics and previous IT experience) – Section 2 (evaluation of the software)– Section 3 (evaluation of the learning experience)

Questionnaire results – section 1 (demographics)

• 8 students completed questionnaire (7 female, 1 male)

• Evenly split between age groups 21-30 and 41-50

• All had access to computers at home and work, including broadband internet

• All were comfortable using word and sending emails

• Limited or no experience in using the Blackboard VLE or online discussion boards.

Questionnaire results – section 2 (software navigation)

Element assessed No. of students

Comments

Ease of navigation around the system

5/8 Not too difficult

3/8 Quite easy

Frequency of mistakes / errors

8/8 Some commonly repeated errors (eg. minimising screen)

Ease of rectifying a mistake

7/8 Noted it was easy to rectify

Writing reports into the discussion board

7/8 Easy to do

Moving between cases and the discussion board

8/8 Easy to do

Questionnaire results – section 2 (image quality)

Element assessed No. of students

Comments

Layout of images (thumbnails/enlarged)

7/8 Appropriate and consistent

Order of images mimics clinical practice

2/8 No single way of viewing images exists, yet cases were still negotiable to all

Image quality – contrast, sharpness, resolution

8/8 Adequate to make a judgement on in a formative setting

4/8 Images could be larger and brighter, benefit from image manipulation tools

Quality of accompanying clinical information

6/8 Sufficient to support the diagnosis

Level of difficulty of cases 5/8 Quite difficult

3/8 Straight-forward

Questionnaire results – section 3 (learning experience)

• 3/8 students felt a little uncomfortable allowing other students to access their work, in case they got it wrong. However 5/8 students were not worried by this.

• All students welcomed the facilitators input and the discussions that it promoted.

• All students felt it would be useful to continue these exercises off campus, with one suggesting it should be voluntary.

Students free comments :Evidence of peer review in progress

 

•Helpful to see other people’s wording and then compare to your own report.

•People phrase thing differently, some people use many words but come to the same conclusion with no additional information.

•Good to see other people’s structure and thought patterns, and useful to see how other people describe pathology.

•I am not used to viewing images in this way, it is a little unsettling but I will get used to it.

•I missed most of the pathology which was disappointing, and a little embarrassing, but I think the viewing conditions take some getting used to.

•I answers wrong, but it is useful to see other people’s opinions and report am uncomfortable at the thought of getting the writing styles.

Students free comments : Concerns about sharing their reports

 

•Helpful to see other people’s wording and then compare to your own report.

•People phrase thing differently, some people use many words but come to the same conclusion with no additional information.

•Good to see other people’s structure and thought patterns, and useful to see how other people describe pathology.

•I am not used to viewing images in this way, it is a little unsettling but I will get used to it.

•I missed most of the pathology which was disappointing, and a little embarrassing, but I think the viewing conditions take some getting used to.

•I am uncomfortable at the thought of getting the answers wrong, but it is useful to see other people’s opinions and report writing styles.

Discussion of key findings -image quality

• Evaluations generally very positive, BUT

• Some cases do not have optimum contrast and resolution – limited by resolution of computer monitor and the type of image file.

• Recent purchase of computed radiography systems by the University will enable importing of high resolution (DICOM) files – these will result in much higher quality images.

• Lack of image manipulation tools - free software package currently being explored (K-PACS)

Discussion of key findings – navigational errors

• Some errors were expected (students new to VLE and discussion boards)

• Some were repetitive, but a new upgrade of the VLE has reduced the opportunity for clicking on the wrong buttons

• A “help file” to be created to assist in navigating around the system and optimising the system prior to use

Discussion of key findings – self and peer assessment

• Students very much valued being able to read each others reports. Clear from their comments that peer assessment is indeed being put into action.

• 3/8 felt a little uncomfortable in posting their reports for others to view

• Following a posting of a definitive report and the tutors comments, statistical review showed that all students had re-visited the cases and each other’s reports, with several posting further comments and queries.

• Facilitators role is believed to be critical to the success of this learning package.

Discussion of key findings –facilitator’s role

• Facilitator is a catalyst – ensuring that students make the most of the learning experience.

• Must be vigilant in checking for further postings or queries

• Facilitators need to develop new skills to manage the process effectively

• Tactful feedback to the group and to individuals is essential to keep students motivated and reduce any feelings of embarrassment or inadequacy.

Discussion of key findings – mandatory or optional?

• In the questionnaire, 1 student indicated that further off campus sessions should be optional.

• In the first off campus delivery, 1 student did not take part. It is not clear if this was the same student who indicated it should be optional.

• If these sessions are not mandatory, then students could opt out of contributing a report, yet could still access the other reports and the facilitator’s feedback.

• Potential inequity and unfairness – reaping the rewards of other people’s hard work.

• The “minimal contributor” is a concept often encountered in other forms of group work, such as problem based learning.

• No easy answers – further monitoring required.

Conclusions and recommendations

• The learning package evaluated within this project appears to be welcomed.

• It is still in transition as new technology becomes available.

• Off-campus delivery is now scheduled for the next clinical period, with further evaluation.

• External evaluation by an expert in the field

– for the student learning package

- to assess whether the learning package can be adapted as an audit / CPD tool for already qualified reporting radiographers

Conclusions and recommendations

• The learning package evaluated within this project appears to be welcomed.

• It is still in transition as new technology becomes available.

• Off-campus delivery is now scheduled for the next clinical period, with further evaluation.

• External evaluation by an expert in the field

– for the student learning package

- to assess whether the learning package can be adapted as an audit / CPD tool for already qualified reporting radiographers

References

Cancer Research UK (2004). Incidence of Cancer in the UK. Latest statistics (2004). http://info.cancerresearchuk.org/cancerstats/incidence/?a=5441. Accessed 13.07.07.

Markus JB, Somers S, O'Malley BP and Stevenson GW. (1990). Double-contrast barium enema studies: effect of multiple reading on perception error. Radiology 175, pp. 155–156.

Neilsen J. (2005). Ten Usability Heuristics, in Heuristics for Interface Design. Posted 2005. www.useit.com/papers/heuristic/heuristic_list.html, accessed 17/06/07.

Thank you for listening

J.Nightingale@salford.ac.uk