Wearable Computing and Human Computer Interfaces
-
Upload
jeffrey-funk-creating-new-industries -
Category
Business
-
view
95 -
download
1
Transcript of Wearable Computing and Human Computer Interfaces
Jeffrey Funk
Division of Engineering and Technology Management
National University of Singapore
Human-Computer Interfaces and Wearable Computing
Thanks to Karthik Nandakumar for the first drafts of these slides in 2012For information on other technologies, see http://www.slideshare.net/Funk98/presentations
What do the Following Trends Mean for Human-Computer Interfaces?
Better ICs, MEMS, wireless transceiversMoore’s LawMore than Moore
Better camerasHigher resolution and sensitivity
Better displaysCheaper, flexible and more durableMore sensitive touchNew forms of touch
Better combinations of these interfacesBetter neural interfaces
A New Generation of Input Interfaces
Neural
Speech
Touch
GestureAugmented Reality
Wearable
That Can Change the Way We Live and Work
Not just consumer applications Assemblers can see drawings Construction workers can see through walls,
wires, and pipes Prospectors can see through the ground Architects can see entire 3D image of building Students can see 3D representation of anatomy,
materials, universe Similar systems could be useful for tourists,
shopping, soldiers, and artists
HCI
Human-Computer Interface (HCI)
• HCI is the technology that connects man and machine
• Robust HCIs are needed to enable ubiquitous computing
We primarily focus on input interfaces
Human Computer
Thoughts
ActionInput
InterfaceAction
Recognition
Task Execution
Understanding
Output Interface
Sensory Perception
Rendering
Traditional Input Interfacesare Disappearing..
Command Line Interfaces (CLI), i.e., keyboard
Batch Interfaces
Graphical User Interfaces
(GUI)
Even God is Interested!
Session Technology
1 Objectives and overview of course
2 How/when do new technologies become economically feasible?
3 Two types of improvements: 1) Creating materials that better exploit physical phenomena; 2) Geometrical scaling
4 Semiconductors, ICs, electronic systems
5 Sensors, MEMS and the Internet of Things
6 Bio-electronics, Health Care, DNA Sequencing
7 Lighting, Lasers, and Displays
8 Human-Computer Interfaces and Wearable Computing
9 Information Technology and Land Transportation
10 Nano-technology and Superconductivity
This is Eighth Session of MT5009
Outline
• Overview
• Speech Interfaces
• Touch Interfaces
• Gesture Interfaces
• Augmented Reality (various combinations)
• Wearable Computing
• Neural Interfaces
Performance of Input Interfaces
• Accuracy: Precision in recognizing the action
• Throughput: Information that can be processed per unit time
• Affordability: Inversely proportional to the cost
• Ease of use: Ease in learning to use the interface
• Sociability: Multi-person interactivity
• Mobility: Size and mass of device, power consumption
• User experience: Subjective perceptions of utility & efficiency
Comparison of Input Interfaces
Key Components of Input Interfaces Human-Computer Input Interfaces
Neural Interfaces
Natural UI
Speech
Micro phone
Neural electrodes/
sensors
GestureTouch
3D Camera
Touch sensor
Tracking & Recognition
Software
Materials/Nanotechnology
Signal Processing Hardware (Semicon-ductors)
Outline
• Overview
• Speech Interfaces
• Touch Interfaces
• Gesture Interfaces
• Augmented Reality (various combinations)
• Wearable Computing
• Neural Interfaces
Speech Interfaces
Key Components
Microphone
Automated Speech Recognition (ASR) and Natural Language Understanding (NLU) Software
Possible methods of improvement are
• Increase Signal to Noise Ratio (SNR) from microphone
• Achieve human-level performance on ASR/NLU tasks
Key dimension that needs improvement is Accuracy
Video on Siri (iPhone 4S)
http://www.youtube.com/watch?gl=SG&hl=en-GB&v=L4D4kRbEdJw
Evolution of Microphone Technology
Electret Condenser Microphone (ECM)
Signal to Noise
Ratio (SNR): 55-58 dB
MEMS Digital Microphone
SNR: 61 dBFlatter frequency response
Smaller size (CMOS fabrication)Can be reflow soldered
Analog Devices, MEMS Microphone Technology, October 2010
MEMS Microphone Technology
Signal to Noise Ratio (SNR) is directly proportional to
• Area of back-plate• Air gap spacing• Applied bias voltage• Mechanical compliance factor (inversely proportional to
stiffness of the diaphragm)
Air gap spacing
Sound wave
Elko et al., Capacitive MEMS Microphones, Bell Labs Technical Journal, 2005
Can MEMS Microphones be Improved?
• Size of microphone depends on area of backplate & gap spacing
• Reducing size will also reduce the quality (SNR), unless it is compensated by other factors
• Wavelength of audible sound waves is > 17 mm, while size of a MEMS microphone is only a few mm; noise level in a microphone is already close to the thermal noise limit
• Dramatic reductions in size or increase in SNR of single MEMS microphones does not appear to be possible
S. Beus, “MEMS mic enables thinner phone designs”, 2005
Microphone Array
Microphone array can mitigate background noise & interference
Courtesy: Audience Inc.
Accuracy based on Microphone Array
1. LOUD project from MIT Computer Science and Artificial Intelligence Laboratory, 2005
2. Microphone Array project in MSR: Approach and Results, Microsoft Research, June 2004
Noise suppression algorithms can increase SNR by 18dB with just 4 microphones in an array1,2
Improvement in SNR Corresponding decrease in Word Error Rate
Automated Speech Recognition (ASR)
“Increase in vocabulary sizes needs exponential increase in computing power due to potential combinatorial explosions”
L. Rabiner, “Challenges in Speech Recognition”, NSF Symp. on Next Gen. ASR, 2003
Automated Speech Recognition (ASR)
“Increase in vocabulary sizes needs exponential increase in computing power due to potential combinatorial explosions”
L. Rabiner, “Challenges in Speech Recognition”, NSF Symp. on Next Gen. ASR, 2003
Only Acceptable in Some Niches
Wor
d er
ror
rate
ASR Accuracy in Text Dictation
• Stand-alone speech interfaces may be useful for tasks like dictation
• Speech as an important modality in multimodal user interfaces (e.g., Microsoft Kinect) may be the future
* http://blogs.msdn.com/b/sprague/archive/2004/10/22/246506.aspx
*
?
For Online TranslationsGoogle’s service translates documents with
inference models that were developed from analysis of a trillion words or 95 billion English sentences.
By 2010, its dataset covered more than 60 languages and could accept voice input in 14 languages and English is sometimes used as a bridge between two different languages when direct translations don’t exist
Siri
Outline
• Overview
• Speech Interfaces
• Touch Interfaces
• Gesture Interfaces
• Augmented Reality (various combinations)
• Wearable Computing
• Neural Interfaces
Touch ScreensMany kindsBut most are
variations of eitherResistive Capacitive (iPhone)
Depend on new materials that are deposited on top of an LCD display
Processors interpret the datahttp://www.youtube.com/watch?
v=FyCE2h_yjxI&src_vid=5fOI-EQCOOQ&feature=iv&annotation_id=annotation_558874
What are the limitations with these touch screens?
Which technologies might contribute towards overcoming these limitations?
CHALLENGE: SENSITIVITY
Source for next 15 slides: Group presentation in Fall 2014
Challenges – Sensitivity
0 1 2 3 4 5 60.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
140.00%
160.00%
180.00%
200.00%
f(x) = 0.138313454391182 x^-1.08178288123264
Sensitivity Level vs Overlay Thickness
Ratiotrendline
Overlay Thickness (mm)
Percentage Change in Raw
Count
*Note: Microprocessor doesn’t recognize capacitance domain but rather it register the change in raw count
Source: FYP Industrial Collaboration (Fischer-Tech and NUS) Project Report 2013 – An Empirical Approach towards Capacitive Touch-Sensing in Functional Plastics
APPLE IPHONEModel Type Overlay Thickness
3GS, 4, 4S Gorilla Glass 1 1.0mm5, 5S Gorilla Glass 2 0.8mm
SAMSUNG GALAXYModel Type Overlay Thickness
S1, S2 Gorilla Glass 1 1.0mmS3 Gorilla Glass 2 0.8mmS4, S5, Note 3 Gorilla Glass 3 0.4mm
1.0mm
0.8mm
0.4mm
Thinner Glass Increases Sensitivity and Use of Smart Phones in Cold Countries
Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com and http://www.corning.com/WorkArea/showcontent.aspx?id=63819
CHALLENGE: DURABILITY
Low Damage ResistanceLow Bending StrengthLow Critical Load Bearing
New Generations of Gorilla Glass: Higher Loads
Through IOX: ion-exchanged glass and new processhttps://www.youtube.com/watch?v=q4ZU7zUxdM8
Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com
*Note: Critical Load is the min. amount of load before radial cracks start to form and propagate
4500
15000
7500
New Generations of Gorilla Glass Have Tighter Strength Distributions
(low probabilities have been eliminated)
Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com
Gorilla Glass 1 Gorilla Glass 2 Gorilla Glass 3
4 MPa 6.25 MPa 6.75 MPa
Greater Strength at all Thicknesses
Source: [Spec sheet download for glass 1, 2 and 3] http://www.corninggorillaglass.com
65
75
85
Gorilla Glass 1
Gorilla Glass 2
Gorilla Glass 3
CHALLENGE: FLEXIBILITY AND CONFORMITY
Achieving Flexibility and Conformity
OLEDs are more flexible than are LCDsBut other parts of the display are not flexible
Transparent conductorExisting Glass
Indium-tin oxide is inflexible and must be replaced with new materialSilver Nano wires?Carbon nanotubes, graphene?
Glassneed thinner glass, which is becoming feasible
Carbon Nano Tubes are More Flexible than Indium-Tin Oxide
Source: http://iopscience.iop.org/1347-4065/53/5S1/05FD04/article
Thinner Glass Leads to Greater Flexibility
Source: www.corning.com/WorkArea/downloadasset.aspx?id=48957
Willow Glass
Thinner Glass Leads to Lower Bend Stress and Failure Probability
Source: www.corning.com/WorkArea/downloadasset.aspx?id=48957 and http://www.corning.com/WorkArea/showcontent.aspx?id=63819
CHALLENGE: LIGHT TRANSMISSION
Read and Input on Both Sides
GamesSurgical Discussions through Glass
LG announceda transparenttelevision (30%)
Do we need one?
Or is that anoxymoron?http://www.extremetech.com/computing/186241-lgs-flexible-and-transparent-oled-displays-are-the-beginning-of-the-e-paper-revolution
Need New Forms of Transparent Conductors
Source: http://iopscience.iop.org/1347-4065/53/5S1/05FD04/article , http://www.beilstein-journals.org/bjnano/single/articleFullText.htm?publicId=2190-4286-4-12
ITO is expensiveand inflexible
Can we use:
Silver Nanowires?
CNTs?
Graphene?
Are Silver Nano-Wires the Best?
At 100 Ohm per Area, Light Transmission of:
• ITO – 91%• Cambrios ClearOhm
(Ag Nanowire) – 98%
OTHER CHALLENGESExisting touch screens require one to look
carefully at screen while touching a specific place
Fingers can easily touch wrong placesTactus offers an overlay to existing
touch screens that facilitates proper location of finger“Bubbles” rise out of the display when fingers touch
the display thus helping fingers find the “right spot”These bubbles are formed using MEMS (micro-
electronic mechanical systems)Studies have found that faster and more accurate
typing are achieved with the Tactus overlay
How the Tactus System Works
Micro-channels are filled with fluid whose refractive index matches that of top polymer layer. Thus, transparency is even across surface. http://www.youtube.com/watch?v=wrSKbTzc4BI 0:40-1:30
Texture Touch DisplaysSensation of texture can provide more information for
usersThis can be done using changes in vibration with
small motors or transparent electrodes (Senseg) that provide
information about texture, etc. www.youtube.com/watch?v=FiCqlYKRlAA (from 0:30-2:00 minute mark)
Another one from Disney: www.washingtonpost.com/
blogs/the-switch/wp/2013/10/08/disney-invents-touchscreen-that-lets-you-feel-textures/
Early applications: 3D modeling or remote surgery can benefit from data on texture of materials or organs
Sensing Force of TouchNext generation Apple (this fall) will detect
how hard user is pressing on screenCan distinguish between light tap and deep
pressNew applications?
Piano-playing app?New types of games?
Applications for Touch ScreensSmart phones for purchasing?
http://www.youtube.com/watch?v=Gg3tmZrwbDs
Smart displays in laboratoriesAlso
Advertising displays at bus stops or MRT stations
Mall information displaysSelf checkout in storesInformation counter in storesFor example, Sony’s AtracTable is
being developed for these applications
Outline
• Overview
• Speech Interfaces
• Touch Interfaces
• Gesture Interfaces
• Augmented Reality (various combinations)
• Wearable Computing
• Neural Interfaces
Components of Gesture Interfaces
Key Components
2D/3D Camera (image sensor)
Tracking, Recognition &
Gesture Understanding
Software
Key dimensions that need improvement are Accuracy, Throughput and Affordability
Working of a 2D Image Sensor
http://www.cameratechnica.com/2011/11/30/five-reasons-you-may-soon-be-shooting-at-iso-50000/
Image Sensor Characteristics• Spatial resolution: Number of pixels
• Temporal resolution: Frames per second
• Image sensor area: Size of the image sensor – area is proportional to no. of pixels and pixel size
• Photometric exposure: light gathering ability of the sensor – depends on the properties of the lens
• “Light available per pixel”: No. of photons incident on a pixel – proportional to photometric exposure and pixel size
• Pixel sensitivity: is proportional to “light available per pixel”, quantum efficiency of photodiodes, and optical efficiency
• Dark Limit & Dynamic range: Ability to detect dim details & bright details in one image – depends on pixel sensitivity and capacity
Improvements in Image SensorsAccuracy
• Higher spatial resolution (no. of pixels)
• Robustness to lighting changes (high pixel sensitivity, low dark limit, and high dynamic range)
• More accurate depth sensing (lower depth error)
Throughput
• Higher frame rate
Affordability
• Smaller pixel size reduces price per pixel
Improvements in Spatial Resolution
T. Suzuki, “Challenges of Image-Sensor Development”, ISSCC, 2010
Number of pixels (resolution) has increased, but image sensor size has not increased because of reduction in pixel size
Year
Pixel Size vs. Sensitivity Tradeoff
CMOS-based image sensors are also expected to follow Moore’s Law in size and cost scaling
T. Suzuki, “Challenges of Image-Sensor Development”, ISSCC, 2010
As pixel size decreases, “light available per pixel” will become less, so sensitivity decreases
Back illuminated CMOS technology provides better trade-off between pixel size and sensitivity than traditional charge coupled device (CCD)-based image sensorsL
igh
t av
aila
ble
per
pix
el (
arb
. un
it)
Camera Technology Improvements
T. Suzuki, “Challenges of Image-Sensor Development”, ISSCC, 2010
Reducing pixel-size (green square) and improving sensitivity (Yellow circle ) miniaturized cameras without reducing quality
Image Sensors vs. Human Eye
Number of frames per second
Spati
al re
solu
tion (
cycl
es
per
degre
e)
Human Eye
Better than Human Eye
Modern cameras are close to human eye in terms of resolution
Skorka & Joseph, “Toward a digital camera to rival the human eye”, J of Electronic Imaging, 2011
Image Sensors vs. Human Eye
Dynamic Range (dB)
Dark
Lim
it (
cd/s
q.m
)
Human Eye
Better than Human Eye
But improvement can be achieved in terms of sensitivity
Pixel sensitivity determines the dark limit and dynamic range of an image sensor
Further Improvements in Sensitivity• Increase quantum efficiency through the design of better
photosensitive materials using nanotechnology
• (e.g., Single Carrier Modulation Photo Detector)
• Increase optical efficiency through a “vertically integrated” layered arrangement as in the human retina
http://www.future-fab.com/documents.asp?d_ID=4926
Cost per pixel of Camera Chips has fallen dramatically
3D Depth Sensing Technologies
* R. Lange, “3D Time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology”, PhD Thesis, 2000
Comparison of 3D Sensing Technologies
Application Range (m)
Depth
Reso
luti
on (
m)
Usable Range for Gesture Interfaces
Microsoft Kinect
• Cost-effective 3D image sensors are now becoming available (e.g., Microsoft Kinect ~ 150 USD)
• Such cameras will further improve the accuracy of gesture UIs
3D Depth Sensing: Interferometry
• Most accurate depth sensing technology (accuracy depends only the wavelength of light)
• Low miniaturization potential and very limited range
3D Depth Sensing: Time of Flight
• Time of flight (ToF) cameras requires processors with high clock speed (3 GHz speed can provide only 4.5 cm depth resolution)
• High miniaturization potential and large range
• Improvements in CMOS technology are likely to very beneficial
3D Depth Sensing: Triangulation
Passive TriangulationActive Triangulation
• Limited range
• Low miniaturization potential
• Depth accuracy decreases with square of the distance
Leap has Generated Excitement
http://www.youtube.com/watch?v=_d6KuiuteIA
Leap uses multiple camera sensors to recognize gesturesWorkspace is about 3 cubic meters. Better sensors will enable larger work spaces. $70 control system that plugs into any computer. MIT’s Technology Review calls Leap, “The most important new technology since the smart phone…”
How about Microsoft’s Kinect? http://www.youtube.com/watch?v=o4U1pzVf9hY
Or wearable ring (each position represents different number)? http://www.youtube.com/watch?v=Gx3zWHS8amA
Replace Cameras with MEMS-based wrist band from Thalmic Labs, called MYO
Gestures are recognized before movement
Muscle activityis monitored with9-axis inertial measurement unit (MEMS)
Other Applications for 3D CamerasUse cameras to
track eye movementsMonitor drivers or
other operators of machines
Help paralyzed people use computers
As cost of cameras fallEye tracking might
become user interface for non-paralyzed
Eye tracking can also be used with Google Glasses (see below)
Source: http://www.economist.com/news/technology-quarterly/21567195-computer-interfaces-ability-determine-location-persons-gaze
3D Cameras and Virtual Reality Can Improve Video Conferencing
Might this finally be the technology that reduces need for travel?
Intel introduced RealSense technology, which gives the cameras in laptops the ability to see and understand depthNotebooks this yearTablets and phones next year?
Eliminates the background, enables better communication with hands
Virtual reality makes the video conferencing even betterVR discussed in 7th sessionAltspaceVR is designed for video conferencing
Outline
• Overview
• Speech Interfaces
• Touch Interfaces
• Gesture Interfaces
• Augmented Reality (various combinations)
• Wearable Computing
• Neural Interfaces
Types of Augmented Reality
Glasses
Phones
How Different from Virtual Reality?
Augmented Reality Supports our understanding of the real world while virtual reality immerses us in a new type of world
Recently Oculus VR was acquired by FacebookImprovements in 3D displays, accelerometers,
gyroscopes, compasses (MEMS) and graphic processors are enabling rapid improvements in VR
How might Oculus VR help Facebook?Will users want further immersion in social
networking……
Returning to Augmented Reality
What do you see through the glassesor Lens?
Handheld devices may be sufficient, particularly if the images are easily integrated with your surroundings
What about a or a virtual oneSupermarket? at a subway station?
http://www.youtube.com/watch?v=yKNSOwLcrkE
What about superimposing the images on a car’s windshield?
Outline
• Overview
• Speech Interfaces
• Touch Interfaces
• Gesture Interfaces
• Augmented Reality (various combinations)• Wearable Computing Main Source: A Rajaraman, B Madhumita,
Mayank Tewari, D Nelson, S K Rao, Wearable Technology Design, Spring 2015
• Neural Interfaces
“Moore’s Law” and “More than Moore” are Making Wearable Computing Economically Feasible
But where will the devices be attached?
Wrist/Arm?
Leg?
Head?
Body?
Where will the Devices be Attached?
Different data can be collected from different parts of the body
Data can be viewed better on some parts of the body
What does this mean for the products and services that will succeed?
Services and software will be an important part of wearable computingNot just hardware!
Lets look at current products and where they are attached
Future projects should probe deeper, providing a better idea about the best places to attach devices
Skull
Google’s Project Glass
Image and information are displayed on the glassesUsers choose which information to display on the glassesChoice controlled by voice, remote control, or maybe
thoughts in futurehttp://www.youtube.com/watch?v=9c6W4CCU9M4 Improvements in ICs, displays, other components are
leading to better performance and cost of glassesBut maybe they won’t help men find a girlfriend
http://www.youtube.com/watch?v=8UjcqCx1BvgUnless you can access data without talking (e.g.,
tilting one’s head, touching the device, or blinking your eyes)
Google Glass in Factories (1)Replace stacks of wiring
instructions with Google Glass display
Test programs are being conducted at Boeing, Daimler, United Parcel, and others
Boeing workers use them to assemble wire-harnesses
When assembler reads out loud coding on wire, correct hole on the electronic version lights up and flashes, providing easy to follow guide
Error rates and assembly time have fallen
http://www.wsj.com/articles/smart-glasses-get-new-look-on-factory-floor-1433301177
Google Glass in Factories (2)Deutsche Post is working with DHL and Ricoh
Warehouse workers process orders ofr parts and equipment by scanning bar codes on cartons with their glasses
Eliminates need for hand-held bar-code scanners and paper invoices
Instructions are relayed through glassesReplace stacks of wiring instructions with Google Glass
displayDaimler uses them on assembly lines
Glasses provide checklists so workers don’t have to memorize or hold the paper lists
Discovered defects are immediately reported through voice-recorded report with photos
Safety and security are issueshttp://www.wsj.com/articles/smart-glasses-get-new-look-on-factory-floor-1433301177
Other Examples
Microsoft’s HoloLensCan be used to play games Or to interact with the world https://
www.youtube.com/watch?v=Qm2gnnyyvEg
Other exampleshttp://www.youtube.com/watch?v=t-m6
YL64lkUhttps://www.youtube.com/watch?
v=fJI8tNG1rbQ
Whole ARM
WHOLE LEG
Can this Analysis be taken Further?What are the costs/prices?
How much will they come done and how fast?What are the benefits and where are they the
largest?Can we quantify the potential benefits from
specific wearables? Do specific sensors (earlier tonight) work with
specific wearables? Which sensors are getting cheaper and better?Can we identify where the largest benefits might
be?Can we use this information to design better
wearable devices
Perhaps More ImportantlyNew forms of software are needed for
wearable computingOperating systemsCloud computingOpen sourceWhat startups will do this and be valued at more
than $1 BillionNew forms of software services are needed
Big dataConsumer InternetWhat startups will do this and be valued at more
than $1 Billion
One example of New Software
Will new forms of passwords be needed for wearable equipment
Fingerprints and iris scans require special equipment, thus increasing the weight of wearable equipment
Ballisocardography is study of body’s movements in response to the activity of the heartBody shifts slightly as heart beatsUnique to individualsShifting Identity, Economist, June 20, 2015, p.
76
Outline
• Overview
• Speech Interfaces
• Touch Interfaces
• Gesture Interfaces
• Augmented Reality (various combinations)
• Wearable Computing
• Neural Interfaces
Neural Interfaces
Key Component
Brain scanning device
Key dimensions that need improvement are Accuracy, Throughput and Affordability
Required improvements in brain scanning technology
Accuracy & Throughput – Higher spatial and temporal resolution
Affordability – Size and better materials
Current State of ArtBest systems enable a person to control a
robot or cursor or type one letter a minute with their “mind”For paralyzed, very expensivehttp://www.youtube.com/watch?v=C7H_M8-dBHc
(0-1 minute)http://www.youtube.com/watch?
v=cDiWFcA0gaw&playnext=1&list=PL7FD931F8953A0F87&feature=results_main (0-2:45 minute)
Accessory for your iPhone$99 device measures your
brain waves. App displays data on phone
http://singularityhub.com/2011/01/07/iphone-accessory-from-xwave-channels-your-brain-waves-to-the-iphone/
SPECT
EEG
1936 1950 1972 19751968
CT Scan
1983
MEG
1991
fMRINIRS
1973
MRI PET
US$2.9M
US$1M-1.5M
US$250K US$2.4M US$0.5M-3M
US$180K- 250K>US$30K
Non-Invasive Brain Scanning
Electro Encephalo Graphy
Magneto Encephalo Graphy
Near-Infra Red Spectroscopy
functional Magnetic Resonance Imaging
EEG & MEG directly measure neuronal activity, NIRS & fMRI measure blood activity
What do EEG & MEG Measure?
Localization of EEG vs. MEGEEGMEG
Where we Are for Resolution (1)
Ideally, a non-invasive technology with high spatial resolution and high temporal resolution is required
Additionally, the technology must be affordable and portable in order to be useful in HCI applications
Gerven, M. v., et al., “The Brain-Computer Interface Cycle”, J. Neural Eng, 2009
Non-invasive
Neuron can fire ~0.1mm (spatial) & ~10 ms (temporal)
Invasive
Where we are, and where we want to be (2)
(in 60 years?)
http://singularityhub.com/2011/01/07/iphone-accessory-from-xwave-channels-your-brain-waves-to-the-iphone/
Invasivetechniques
Spatial Resolution Improvement
While spatial resolution is important for accuracy, high temporal resolution is also critical for user interfaces
R. Kurzweil, “The Singularity is Near”, 2005
fMRI
EEG Challenges
Key limitation: Poor spatial resolution
Increasing number of EEG electrodes may provide limited improvement in spatial resolution and higher SNR
J. Malmivuo, “Comparison of the Properties of EEG and MEG”, Intl J of Bioelectromagnetism , 6 (1), 2004
MEG Challenges
Baranga, A. B.-A. (2010). "Brain's Magnetic Field: a Narrow Window to Brain's Activity". Electromagnetic field and the human body workshop, (pp. 12).
fT
MEG: Improvements in Millimeter-scale Atomic Magnetometer
Target: ~100fT and <100Hz
J. Kitching, et al., “Uncooled, Millimeter-Scale Atomic Magnetometers”, IEEE Sensors 2009 Conference (pp. 1844-1846)
A: 2004B: 2007C: 2007D: 2009
Invasive Techniques
Chips are implanted intoa person’s brain
Source: Stevensen I, Kording K 2011, How Advances in Neural Recording Affect Data Analysis,Nature Neuroscience 14 (2): 139-142
Improvements in Invasive Techniques
Source: Stevensen I, Kording K 2011, How Advances in Neural Recording Affect Data Analysis,Nature Neuroscience 14 (2): 139-142
But the improvements are probably not occurring fast enough
At current rates of improvement, in 220 years we will be able to simultaneously record all 100 billion neurons
On the other hand, maybe we don’t need to record all of
them simultaneously because we will find ways to interpret the data
Or maybe we will find better non-invasive techniques
Conclusions
• Rapid improvements are occurring in HCI
• Most of these improvements are being driven by improvements in ICs or MEMS (reductions in scale)• Speech recognition (microphones, processors)
• Gesture interfaces (cameras)• Neural interfaces (electrodes)
• Other improvements are being driven by creation of new materials• Touch screens
Pictorial Representation of the Drivers
Microphone
Multitouch Sensors
3D camera & motion sensors
Haptic Devices
Neural Sensors
Speaking
Touch Events/
Gestures
Body/head movements
Free hand gestures
Facial expression
Eye gaze
Hand pressure
Brain activity
Speech Recognition &
Language Understanding
Touch/Gesture Recognition & Understanding
Neural Signal Analysis &
Understanding
User Interface System Design & Integration
Multimodal Fusion
Human Factors
Engineering
Context Aware
Services
Conclusions
• Ubiquitous computing requires new HCI paradigms
• Natural and Neural Interfaces are the future of human-computer input interfaces
• Touch interfaces have already diffused into the mainstream; speech and gesture interfaces are becoming more accurate and affordable
• Neural interfaces requires development of more accurate, cheap, and portable sensors
• Numerous entrepreneurial opportunities are available both in technology development & customization
One-Page Write-ups4 one-page write-ups on topics related to technologies covered
in sessions 4 through 10 (20% of your grade)If you had two months to investigate one of these topics, what would
you do? What types of data would you gather and how would you gather the
data? Who might you interview? Topics are listed at end of these slides, of each session, and in
assessment section of IVLE
2 one-page write-ups on topics related to technologies covered in group presentations (10% of your grade)Topic should be main points of presentationpropose a different method of analysis than the group did
It should not take you longer than 2 hours to do each assignment
Be Careful!The one-page write-ups on topics (related to technologies covered
in sessions 4 through 10) are different than the 2 one-page write-ups on (group presentations)They require very different answers
Even for one-page write-ups on topics related to technologies covered in sessions 4 through 10Each question is different because cost and performance depends on
different things for each technologyEach question is also different because amount of information is different
If technology has been commercialized, there is more info about technology – costs, performance, benefits, needs
If it has not been commercialized, there is less informationSome technologies depend more on improvements in components than do othersThus, the answers will be different
Session 8 Topics for Write-upsGoogle glass How would you assess the costs and benefits of
google glass and when it will become widely used in Singapore (>10% of Internet users)?
Gesture interface: How would you assess the costs and benefits of gesture interfaces and when they will become widely used in Singapore (>10% of internet users)?
Health data recorded with wrist device: How would you assess the costs and benefits of recording health data with a wrist device and when it will become widely done in Singapore (>10% of Internet users)?
Augmented reality with cameras and phones: How would you assess the costs and benefits of augmented reality with cameras and phones and when it will become widely used in Singapore (>10% of Internet users)?