Post on 31-May-2020
Virtual Reality in Medicine and Surgery
Duncan Gillies
Department of Computing, Imperial College London
Virtual Reality in Medicine and Surgery Lecture 1 Slide 1
The History of Virtual Reality - 1615
“Then Don Quixote leaned forward and began to turn thepin in the horse’s head. He fancied that he was rising inthe air, and that he was sailing right up to the sky. ”
Virtual Reality in Medicine and Surgery Lecture 1 Slide 2
The History of Virtual Reality - Flight Simulation
• 1909 Mechanical Simulators• 1929 Concept for electronic simulators• 1945 Point light source projection
(simulating runway lights)• 1955 Video Based systems
Virtual Reality in Medicine and Surgery Lecture 1 Slide 3
The History of Virtual Reality - 1960s
Motorcycle ride throughBrooklyn
A Cinematographic Experi-ence including:• 3D Imagary• Contextual Smells• Stereo Sound• Vibration• Wind in the hair
Virtual Reality in Medicine and Surgery Lecture 1 Slide 4
The History of Virtual Reality - 1970s
• Film Special effects
• Video arcade games
• Bend Sensing Glove
• Data Glove
• Polhemus Tracker
Virtual Reality in Medicine and Surgery Lecture 1 Slide 5
The History of Virtual Reality - 1980s
• See through head mounteddisplay(Callahan (MIT) 1983)
• Commercial head-mounteddisplays(VPL Research and Autodesk1989)
• Term ”Virtual Reality” coined(Jaron Lanier VPL 1989)
• SGI Reality Engine (1989)
Virtual Reality in Medicine and Surgery Lecture 1 Slide 6
The History of Virtual Reality - 1990s
• First Applications of VirtualReality in medical teaching:
• Colonoscopy Simulation• Upper GI tract endoscopy
simulation
• Improved general purposehaptic devices (Sensable)
• First attempts at simulation oflaparoscopy
Virtual Reality in Medicine and Surgery Lecture 1 Slide 7
Types of VR Systems
• Window on World Systems• Hands on Virtual Worlds• Head Tracking Systems• Immersive Systems• Telepresence• Mixed or Augmented Reality
Virtual Reality in Medicine and Surgery Lecture 1 Slide 8
Window on the World
“One must look at a display screen, as a window throughwhich one beholds a virtual world”
Ivan Sutherland 1965
• Conventional monitor
• Realistic Sound
• Special input devices
In reality just simple computer graphics, but stillappropriate for many medical applications.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 9
Window on the World
• Suitable for certain types ofmedical visualisation - forexample virtual colonoscopy
• Given special hardware it can beused for simulating somemedical procedures
• 3D systems with shutter glassesare a possible extension
Virtual Reality in Medicine and Surgery Lecture 1 Slide 10
Window on the World
One problem in creatingwindows on the world forsurgery is designing realistichaptic devices.
They are expensive and areapplication specific.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 11
Hands on Virtual Worlds
• Stereo displays can beconstructed that allow theuser to put his own handsinto the virtual world,
• It is possible to interact withthe image through a generalpurpose haptic device.
• These have been applied tosimulating laparoscopicprocedures.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 12
Fish Tank Systems
Tracking a user’s head position provides improvedrealism.• Using simple stereo devices there is one correct
position from which to view a scene.• By tracking the user’s head, and computing the
stereo image pair accordingly a more accurate 3Dimage can be created.
• No major applications in Medicine as yet.
When applied to the limited spacearound a simple screen this is some-times termed “Fish tank virtual real-ity”
Virtual Reality in Medicine and Surgery Lecture 1 Slide 13
Immersive Systems
Head mounted displays are an alternative to headtracking systems. They provide an immersive experienceby blocking out the real world.
• There is no restriction on usermovement.
• They are relativelyinexpensive and can havegood 3D visual quality.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 14
Immersive Systems
However, they have rarely, if ever, been used for anyserious medical or surgical purpose because of manydrawbacks:
• Restricted field of view withno peripheral vision
• Unnaturally close image• Isolation and fear of real
world events• Uncomfortable to wear
Virtual Reality in Medicine and Surgery Lecture 1 Slide 15
Immersa Desks
• These devices are essentiallyfish tank displays, but arescaled up and have otherfeatures.
• Some have draftingcapabilities, and are wellsuited to applications inarchitecture.
• They have not been usedwidely in medicine.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 16
Cave Installations
• A room with projectionon the walls floor andceiling.
• Provides completevisual immersion.
• Driven by a group ofpowerful graphicsengines.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 17
Cave Installations
• Provides highresolution and largefield of view with lightweight shutter glassesfor stereo viewing.
• Real and virtualobjects can be mixed.
• A group of people caninhabit the spacesimultaneously.
Medical applications couldinclude simulating operat-ing theatre dynamics, butthis has not been tried.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 18
Cave Installations
Cave installations are unlikely to be used extensively inmedical applications because of various drawbackswhich are:• they are very expensive (approximately 1 million
pounds);• they require a large amount of permanent physical
space;• the projector calibration must be maintained;• real physical objects cannot be easily incorporated
with the graphical objects.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 19
Telepresence
Sensors and manipulatorsare remotely controlled by ahuman operator.
Some spectacular success,such as the exploration ofthe Titanic.
Applications include fire fighter robots and unmannedmilitary vehicles.
Proposed for endoscopic or robotic surgery, but not reallyrequired in practice.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 20
Applications of VR in Medicine and Surgery
• TrainingPopular and feasible
• Assessment and CertificationGaining acceptance, feasible
• Telepresence (Interaction with remote environments)Popular but not very useful
Virtual Reality in Medicine and Surgery Lecture 1 Slide 21
Architecture of a Medical VR System
Virtual Reality in Medicine and Surgery Lecture 1 Slide 22
Geometric (Anatomical) Modelling
• Geometric modelling can be used to reproduce viewsof patient anatomy.
• It is used to create walk-throughs of the peritoneum,lungs colon etc.
• Geometric models can be derived from scan data.
As an example let’s look at virtual colonoscopy.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 23
Extracting the Colon Geometry
Virtual Reality in Medicine and Surgery Lecture 1 Slide 24
Creating the walk-through
Virtual Reality in Medicine and Surgery Lecture 1 Slide 25
Walking Through
The images are very pretty:
However the texture colour and shading are fake, andthere are reconstruction errors:
Virtual Reality in Medicine and Surgery Lecture 1 Slide 26
Limitation of Geometric Models
• Triangulated models areeasy to obtain, but aredifficult to deform.(large number of triangles)
• They are suitable for rigidstructures.(larynx, lung, trachea etc)
• They are unsuitable foranimating deformableorgans.(colon, GIT, gallbladder etc.)
Virtual Reality in Medicine and Surgery Lecture 1 Slide 27
Quadratic Gometric Models
• The colon can be effectivelymodelled by a quadratic (sphere)model, which offeres manyadvantages:
• Compact representation• Faster to render• Easy to deform to simulate
spasm and inflation
• However they are a less accuraterepresentation of real anatomy.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 28
Higher Order Geometric Models
• Cubic patch surfaces are good for representingcomplex smooth shapes
• The shape is controlled by a small number of controlpoints (or knots) which can be moved to createanimations.
• However, they are harder to match to real anatomicaldata.
Overall there is a trade off:• Polygon surfaces (linear) can match anatomical
geometry accurately, but are hard to deform.• Quadratic and cubic surfaces are easy to deform and
manipulate, but are anatomically less accurate.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 29
Building a Simlation
• Geometric models can achieve high accuracy, butthey are only one part of a simulation system.
• Typically in a medical application the user needs tomanipulate the underlying geometry - for example incolonoscopy or laparoscopy.
• There are major difficulties in simulating both themechanicl behaviour of the medical instruments andthe interaction with the human organs.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 30
Modelling a flexible colonoscope
Virtual Reality in Medicine and Surgery Lecture 1 Slide 31
The Flexible Colonoscope in the Colon
In practice it is not uncommon to find a colonoscopeconfigured like this:
Does twisting the shaft tighten the loop or rotate theimage?
Virtual Reality in Medicine and Surgery Lecture 1 Slide 32
Variety is the Spice of Life
Unfortunately a colonoscope can end up in manyconfigurations:
Virtual Reality in Medicine and Surgery Lecture 1 Slide 33
Problems in modelling Colonoscope Physics
• What happens when you push the colon wall?• What dynamic properties does the endoscope have?• What frictional forces are acting?• The colon is surrounded by the small intestine which
is also deformable, but how does it deform?• etc.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 34
Behavioural modelling in Colonoscopy
However, one major factor works in favour of colonoscopysimulation:
You cannot see how the colon is being deformed by thecolonoscope.
This means we can use behavioural instead of physicalmodels:• Consultants can describe behaviours that often
occur in colonoscopy (paradoxical behaviour)• Simulating the behaviour is easier than simulation
the underlying physics.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 35
Soft Tissue Deformation
Considerable effort has gone into modelling soft tissuedeformation over the last 30 years. Most approaches arebased on the finite element method.
A complex shape is epresented by a small number ofsimple elements, in 2D:
Virtual Reality in Medicine and Surgery Lecture 1 Slide 36
Soft Tissue Deformation
There are a variety of elements that can be used for 3Danalysis:
Virtual Reality in Medicine and Surgery Lecture 1 Slide 37
Finite Element Modelling of Laryngoscopy
• Geometry must be extractedfrom scan data.
• Boundary conditions must bedefined. (attachment pointsand rigid parts)
• Material properties must bespecified.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 38
Material Properties of Soft Tissues
• The finite element method wasdesigned for engineeringapplications where the materialshave known uniform properties(eg steel structures).
• Soft tissue cannot becharacterised in this way,though good approximatemodels have been developed.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 39
More difficulties in soft tissue modelling
• Data on soft tissues deformation often comes fromcavadars. This data does not represent the in vivobehaviour.
• Other effects come into play. For example whencompressing the tongue initially the blood issqueezed out. After that the muscle becomesconsiderably less compressible.
• More accurate finite element models cannot be usedin real time animation.
Virtual Reality in Medicine and Surgery Lecture 1 Slide 40
Laparoscopic Surgery Simulation
• Laparoscopic surgery simulation has attracted greatcommercial interest over the last 20 years. It isperceived as the most important application ofvirtual reality in surgery.
• Unfortunately however it poses the hardestmodelling challenges:
• Interaction of several systems:BILE ⇔ GALL BLADDER ⇔ SURROUNDING ORGANS
• Cutting tissue, bleeding, flowing bile &c.
• Although it is possible to create plausible graphicsdoes this really represent the real proceure?
Virtual Reality in Medicine and Surgery Lecture 1 Slide 41
Visually Appealing Laparoscopy Simulation
Virtual Reality in Medicine and Surgery Lecture 1 Slide 42
Summary
The modelling problems cover a wide spectrum:
from: No deformation (easy)BronchoscopyColonoscopyUpper GI endoscopyOther endoscopic proceduresLaryngoscopyLaparoscopic Cholecystectomy
to: Large and Complex Deformations (very difficult)
Virtual Reality in Medicine and Surgery Lecture 1 Slide 43
Validation Issues
• Creation of convincing simulations is now possible,but there is no real way of assessing how accuratelythey reproduce the real procedures.
• On the one hand we know that the underlyingmodelling must be approximate, and it is possiblethat using computer simulation might induce badhabits
• On the other hand as training methods for manualskills they could be just as effective - if not better -than older methods
Virtual Reality in Medicine and Surgery Lecture 1 Slide 44