Artificial visionTehisnägemine

Mihhail Šubin

Tallinna Tehnikaülikool, 2010

Contents• Historical facts• Different approaches• Examples, technical details• Conclusions• Used materials

Some historical facts

• Cortical implants– William Dobelle

• Retinal implants– Mark Humayun

Dobelle’s cortical implants

• Inspired by Giles Brindley’s research (1968)• 1970-1972. Cortical stimulation of 37 sighted

volunteers• 1972-1973. Stimulation of visual cortex of three blind

volunteers who were temporarily implanted for a few days

• 1974-1978. Four blind volunteers implanted (one retained implant for 3 months, one for 14 years, two for over 20 years)

(continued)

• 2002. Eight people received implants on commercial basis.

• Total of 16 people implanted on commercial basis, last in y. 2005.

• W. Dobelle died in y. 2004, soon after this Dobelle Institute in Lisbon was closed. Development of Dobelle brain implant is continued by Stony Brook University and Avery Biomedical Devices Inc.

Retinal implants• Proof of principle demonstrated in early 90-s by Mark

Humayun• 1998. Second Sight Medical Products, Inc. Was

founded• 2002-2004. 6 people implanted with first generation

implant (16 electrodes). 5 of them still use the device in their homes today.

• 2009. Second Sight announced that the U.S. Food and Drug Administration (FDA) has granted approval for up to 20 people who are blind or who have severely impaired vision to participate in the ArgusTM II Retinal Implant feasibility study in the U.S.

Different approaches

• Cortical (brain) implant • Retinal implant• Other options?

Brain implant approach

• William Dobelle– Surface implant– Electrodes mapped after implantation– Relatively high current needed– Results introduced (several working

systems)

(continued)

• Richard Normann– Utah electrode array (intra-cortical)– Lower stimulation currents– More precision (small groups of neurons

can be stimulated)– Still no working prototype for human

Retinal implant approach• Chip with array of electrodes on the surface of retina

– Stimulates retinal nerve cells– Receives data by wired or wireless (radio/optical) channel– Electrode mapping should be 1:1 (unlike cortical implants

where electrodes should be remapped to produce logically ordered “picture”)

• Chip mounted outside the eyeball with only electrodes connected to retina– No optical data channel option– Less problems with excess heat

Other options

• Optic nerve stimulation

• Non-intrusive approaches– Sonic vision– Tactile vision

Examples, technical details

• Dobelle’s Artificial Vision system• Utah intra-cortical electrode array• vOICe• Tactile systems

Technical detailsDobelle’s artificial vision system

http://biomed.brown.edu/Courses/BI108/2006-108websites/group03retinalimplants/multimedia/article.pdf

Array of electrodes: drawing and x-ray image

Phosphene map in visual space (electrodes matrix implanted to the right occipital lobe produces

phosphenes in left visual field)

• Frame rate 1 to 10 fps (best results @ 4 fps)• System needs to be calibrated before use (stimulation current

levels may vary)• Image from camera is processed

– To map electrodes in more-or less logical order

– To provide more contrast image (e.g. edge detection)

• Phosphene map is stable (stimulation of certain brain area produces a phosphene in certain spot in the visual field)– Phosphene map moves in the visual field following view direction

(eyes tracking system is needed to stabilize its position)

• Active phosphenes flicker. No color.• Some of the patients experienced seisures while using the

system.• One of the latest patients could even drive a car using 144

electrode implant (72+72 at each hemisphere)

Utah intra-cortical electrode arrayhttp://www.bioen.utah.edu/cni/projects/blindness.htm#program

Electrode array used to find out how nail length influences stimulation results

• Biocompatibility research shows good results – 2-5 uM thick capsule forms around each electrode

• Implantation technology is important– Best results achieved when array is rapidly inserted into

cortical tissues (in 200 uS)

• At first electrode arrays are used on animals to record electrical activity of neurons while visual (or auditory*) stimulation is applied.

• Next stage is behavioral experiments

*auditory cortex is chosen for behavioral experiments because of the ease of providing auditory stimuli

vOICe Auditory Displayhttp://www.seeingwithsound.com/etumble.htm

Original camera image (left) and spectrographic reconstruction from vOICe “visual sounds” (right)

• Brightness corresponds to amplitude, position to frequency

• Technically proven effective resolution up to about 4000 pixels (voicels) – limited to some 1000 to 4000 pixels maximum due to

limitations in human hearing (32 by 32 up to a 64 by 64).

• 32 by 32 pixel resolution is more than enough to get recognizable images

• Frame rate from 1 to 8 fps (depends on resolution)• It is proven that auditory stimulation can cause

excitation of visual cortex, especially for blind people, but vOICe usability is still highly individual– Some patients claim that after long term training, adaptation

and practice they actually see the image, and the sound produced by the system turns to barely noticeable background noise

• Some sound masking occurs when using the system• Image enhancement algorythms (edge detection,

high contrast, negative image) can be used to improve perception

• 1st publication in 1992, system available since 1998

Other options, such as tactile vision substitution systems (TVSS) are also being developed

• Veresk (http://www.tactilevision.ru/english/index.php?id=phil_device)

– Electrotactile– 1000 electrodes fixed around the patients torso– Electrode spacing is currently 8mm

• Numerous electrotactile, mechanotactile and thermotactile systems were developed and tested over time

Conclusions• Dobelle’s system is proven to work, but looks frightening.

Normann’s research is not yet resulted in any practical application, but still seems more solid and promising.

• Retinal implants seem to have become more popular (than cortical ones) over last 5-10 years.

• Non-invasive solutions, like sonic vision systems, look perfect (if they really work as described). They also look like the best choice for people who were born blind.

• Pixelated vision simulators demonstrate how human brain can adapt to lower resolution. Other tests and experiments confirm great flexibility and adaptability of human brain (which is an advantage to be used in implementation of artificial vision system).

List of used materials• http://www.seeingwithsound.com/dobelle.htm• http://www.seeingwithsound.com/etumble.htm(vOICe and Dobelle's brain implant comparison)

• http://www.seeingwithsound.com/winvoice.htm (vOICe demo)

• http://www.wired.com/wired/archive/10.09/vision_pr.html (Article in Wired - Dobelle/Normann/Humayun)

• http://www.bioen.utah.edu/cni/projects/blindness.htm (Utah Electrode Array - Richard Normann)

• http://www.2-sight.com/SSMP_ARVO_2009.pdf (Second Sight retinal implant approval by FDA)

• http://biomed.brown.edu/Courses/BI108/BI108_1999_Groups/Vision_Team/Vision.htm

(cortical implants in general)

• http://www.nidek-intl.com/artificial_vision.html#topics• http://biomed.brown.edu/Courses/BI108/2006-108websites/

group03retinalimplants/multimedia/article.pdf(Dobelle’s article)