Ramesh Raskar Media Lab, MIT Cambridge, MA Second Skin.
-
date post
20-Dec-2015 -
Category
Documents
-
view
227 -
download
1
Transcript of Ramesh Raskar Media Lab, MIT Cambridge, MA Second Skin.
Ramesh RaskarMedia Lab, MIT
Cambridge, MA
Second Skin
Second Skin and RFIG
• Second Skin– Sensing and Actuation
• Every mm at every ms– Impercetible, natural environment
• Overcoming the ‘Dark side of VR’
• RFIG– Marker technology– Marker= RFID + photosensor (not barcode)– Locator= Projector (not camera)– Durable (10+ yrs), long range, imperceptible (zero area)
• Think inverse
Vicon Motion Capture
High-speed IR Camera
Body-worn markers
Medical Rehabilitation Athlete Analysis
Performance Capture Biomechanical Analysis
‘Motion’ Capture ?• Building a real time human model
– Dense sampling over surface– Geometry with Id at every millisecond at every milimeter– Bio parameters
• Getting intimate– Cameras ..– Wearables– Second Skin (Sensor suit)– Tapping inside
• Close the loop in bio-I/O– Remote monitoring: Elderly care, training– Robot observation:learning, worker safety– Feedback for biomech/neuro interfaces
Tagged Books in a Library Id : List of books in RF range
No Precise Location Data Are books in sorted order ?
RF Tag + Photosensor
Conventional RF tag
Photo-sensing RF tag
READER
Micro Controller
RF Data
Light
Memory
Photosensor
Computer
READER
Micro Controller
RF Data
Memory
Conventional RFID
Computer
READER
Projector Micro
Controller RF Data
Light
Memory
Photosensor
Computer
Projector + Photo-sensing RF tag
Pattern
MSB
Pattern
MSB
Pattern
MSB-1
Pattern
MSB-1
Pattern
LSB
Pattern
LSB
For each tag
a. From light sequence, decode x and y coordinate
b. Transmit back to RF reader (Id, x, y)
For each tag
a. From light sequence, decode x and y coordinate
b. Transmit back to RF reader (Id, x, y)
00 11 11 00 00 X=12
X=12
Visual feedback of 2D position
a. Receive via RF {(Id1,x1,y1), (Id2,x2,y2), …}
b. Illuminate those positions
a. Receive via RF {(Id1,x1,y1), (Id2,x2,y2), …}
b. Illuminate those positions
Siggraph 2004
Where are products about
to expire ?
Where are products about
to expire ?
Find tag location using
handheld Projector
Find tag location using
handheld Projector
Photosensing
Wireless Tags
Photosensing
Wireless Tags
Many geometric ops Many geometric ops
R F I
R F I D
Interactive stabilized projection
Interactive stabilized projection
(Radio Frequency Id & Geometry)
G
Siggraph 2004Siggraph 2004
Mitsubishi Pocket Projector
AR with Photosensing RFID and Handheld Projector
Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines‘iLamps: Geometrically Aware and Self-Configurable Projectors’,
SIGGRAPH 2003
AR Issues
• Preprocessing:– Authoring
• Runtime:– Identification: Recognition of objects
• Using markers and visual tags– Registration: Finding relative pose of display device
• Dynamic estimate of translation and rotation• Render/Warp images
– Interaction:• Widgets, Gesture recognition, Visual feedback
AR Issues
• Preprocessing:– Authoring
• Runtime:– Identification: Recognition of objects
• Using markers and visual tags– Registration: Finding relative pose of display device
• Dynamic estimate of translation and rotation• Render/Warp images
– Interaction:• Widgets, Gesture recognition, Visual feedback
RFID ?
AR Issues
• Preprocessing:– Authoring
• Runtime:– Identification: Recognition of objects
• Using markers and visual tags– Registration: Finding relative pose of display device
• Dynamic estimate of translation and rotation• Render/Warp images
– Interaction:• Widgets, Gesture recognition, Visual feedback
RFID
Photosensing RFID
Projector for visual feedback
Inside of Projector
The Gray code pattern
Focusing Optics
Gray code Slide
Condensing Optics Light Source
Tag
2D Location 3D Location
Y data
X data
Y data
X dataX2 data
Pattern
MSB
Pattern
MSB
Pattern
MSB-1
Pattern
MSB-1
Pattern
LSB
Pattern
LSB
For each tag
a. From light sequence, decode x and y coordinate
b. Transmit back to RF reader (Id, x, y)
For each tag
a. From light sequence, decode x and y coordinate
b. Transmit back to RF reader (Id, x, y)
00 11 11 00 00 X=12
X=12
Imperceptible Tags under clothing, tracked under ambient light
Towards Second SkinCoded Illumination Motion Capture Clothing
• 500 Hz with Id for each Marker Tag• Capture in Natural Environment
– Visually imperceptible tags– Photosensing Tag can be hidden under clothes– Ambient lighting is ok
• Unlimited Number of Tags– Light sensitive fabric for dense sampling
• Non-imaging, complete privacy• Base station and tags only a few 10’s $
• Full body scan + actions– Elderly, patients, athletes, performers– Breathing, small twists, multiple segments or people– Animation Analysis
Second Skin and RFIG
• Second Skin– Sensing and Actuation
• Every mm at every ms– Impercetible, natural environment
• Overcoming the ‘Dark side of VR’
• RFIG– Marker technology– Marker= RFID + photosensor (not barcode)– Locator= Projector (not camera)– Durable (10+ yrs), long range, imperceptible (zero area)
• Think inversehttp://raskar.info
Acknowledgements• MERL
• Jeroen van Baar, Paul Beardsley, Remo Ziegler, Thomas Willwacher, Srinivas Rao, Cliff Forlines, Paul Dietz, Joe Marks, Darren Leigh
• Office of the Future group at UNC Chapel Hill• Greg Welch, Kok-lim Low, Deepak B’padhyay, Aditi Majumder, Michael Brown,
Ruigang Yang• Henry Fuchs, Herman Towles• Wei-chao Chen
END
END
END
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Ramesh RaskarMitsubishi Electric Research Labs (MERL)
Cambridge, MA
The Poor Man’s Palace:Special Effects in the Real World
Special Effects and Virtual Worlds
– Photorealism around us ? Stays on screens
– Does it affect daily life in real time ? Unlike other fields
Changing Appearance
Changing Virtual Illumination
Special Effects and Virtual Worlds
– Photorealism around us ? Stays on screens
– Does it affect daily life in real time ? Unlike other fields
– Fusion: real world with graphics• Next big challenge in CG/Second Life ?• Believable, seamless co-existence
Changing Appearance
Projector Projector
Virtual light source
Changing Virtual Illumination
Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)
– Preprocessing• Scan 3D object and create model• Roughly align projector(s)• Calibrate by finding pose
– Run-time• Render images of 3D model• Warp/Correct
Virtual Motion
Virtual Motion
Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)
Dynamic Augmentation
Projecting on Tracked Objects
d ( x )2
k ( x ) cos( p )
Radiance Adjustment
Ip (x, p) =
I
d
, k ( x ) > 0L ( x, )
Virtual Real
Intensity correction
Desired radiance
Pixelintensity
Reflectance
Virtual Reflectance Virtual Illumination
InteractionVirtual Motion
ShaderLamps
www.ShaderLamps.com
Poor Man’s Palace
Poor Man’s Palace
Maya: World is an Illusion
• Fake World– We all live in one
• Social Issues– Real-life Fakes
• Not just photos and videos but surroundings– Privacy
• X-reality/AR/Virtual Worlds – Delivers years of CG/Sensors/Robotics research into the real world– Time and Space shifting with non-linear distortions
• Complex Reflectance – Specular or arbitrary BRDF surfaces– View-dependent appearance
• Participating Media – Simulating or in presence of smoke, fog
• Complex Geometry– Spaghetti
• Motion – Animation of real surfaces
• NPR, Distortions, Perceptual factors– Great thesis topics ..
Beyond Gouraud Shading of White Objects
Pieces of the PuzzleActuated Surfaces
Actuated Workbench [Pangaro, Maynes-Aminzade, Ishii UIST 2002]
Displays Contenders
OrganicLED
LightEmittingPolymers
E-Ink
Recap
• Special Effects in Real World– Photorealism yet to impact daily life– Poor Man’s Palace
• Spatial Augmented Reality– Untethered solution for fusion– Geometry, Photometry, Id– Sense, Control, Compensate– Projectors, RFID, Sensors
• Open Problems– All senses: haptic, olfactory, proprioception– Natural phenomenon, complex BRDF, other displays
• Next Challenge: Photorealistic AR around us