"Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL
-
Upload
embedded-vision-alliance -
Category
Technology
-
view
18.036 -
download
1
Transcript of "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL
![Page 1: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/1.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Using Vision to Enable Autonomous Land, Sea, Air,
and Space Vehicles
Larry MatthiesComputer Vision Group
Jet Propulsion LaboratoryCalifornia Institute of Technology
Gale Crater
Copyright 2016 California Institute of Technology. U.S. Government sponsorship acknowledged.
![Page 2: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/2.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Application Domains andMain JPL Themes
Land: all-terrain autonomous mobility; mobile manipulation
Sea: USV escort teams; UUVs for subsurface oceanography
Space: assembling large structures in Earth orbit
Air: Mars precision landing; rotorcraft for Mars and Titan; drone autonomy on Earth
![Page 3: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/3.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Basic Taxonomy ofPerception Capabilities and Challenges
Capabilities• Localization
– Absolute, relative• Obstacle detection
– Stationary, moving– Obstacle type– Terrain trafficability
• Other scene semantics– Landmarks, signs,
destinations, etc– Perceiving people and their
activities• Perception for grasping
Challenges• Sensors for observability• Fast motion• Lighting conditions
– Low light, no light– Very wide dynamic range
• Atmospheric conditions– Haze, fog, smoke– Precipitation
• Difficult object/terrain types– Featureless, specular,
transparent– Obstacles in grass; water, snow,
ice; mud• Computational cost vs processor
![Page 4: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/4.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyLocalization
• Relevant use cases:– Wheeled, tracked, legged vehicles
indoors and outdoors– Drones– Mars rovers– Mars landers
• Key challenges:– Appearance variability: lighting,
weather, season– Moving objects– Fail-safe performance
• Examples I’ll describe:– Day/night relative localization with IMU,
leg odometry, and NIR active stereo for DARPA LS3 program
– Map relative localization for Mars precision landing
1976 Viking174 x 62 mi
• Some central themes:– Fusion of IMU with vision or lidar
now commonplace for relative localization
– Map matching is a very active topic for absolute localization
![Page 5: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/5.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Day/Night LS3 Relative Navigationwith Visual, Inertial, and Leg Odometry
• Array of NIR LEDs with different lenses/power to achieve more uniform illumination with dense stereo out to ~ 4.5 m
SENSORS:1) Bumblebee Stereo (1024x768)2) Tactical-grade IMU (600Hz)3) Optional: Nav-grade IMU (600Hz)4) Optional: Leg Odometry (200Hz)
![Page 6: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/6.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Day/Night LS3 Relative Navigationwith Visual, Inertial, and Leg Odometry
asphalt dirt road snow forest
- Position error over a moving window of 50m- Position error < 0.5m 95% of runs- Position error < 0.75m 100% of runs
![Page 7: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/7.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Day/Night LS3 Relative Navigationwith Visual, Inertial, and Leg Odometry
• How far can the lookahead scale with illuminators?
• Can it work at night with thermal images (stereo or monocular) – with more noise, motion blur, and rolling shutter readout?
![Page 8: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/8.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Mars Airbag-based Landers (1997, 2004):Horizontal Velocity at Impact
impactvelocity
impactvelocity
impactvelocity
Pathfinder
Pathfinderwith TIRS
Pathfinderwith TIRS ���and DIMES
![Page 9: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/9.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Descent Image Motion Estimation System for 2004 Mars Landers
• Horizontal velocity estimation during last 2 km of descent
AI1
AI2
I1qG
G
g
I2qG
I1
I2
vh11, vh12
AI3
vh21, vh22
I3qG
I3
~ 20 sec with 20% of 20 MHz RAD6000 flight computer
![Page 10: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/10.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Map-Relative Localizationfor Mars Precision Landing
Backshell Separation
Powered Descent
Sky Crane
Flyaway
Prime MLEs
Radar Data Collection
Divert Maneuver
Safe Target Selection
Lander Vision System
TRN increases the ���probability of safe landing
Hazards in Landing���Ellipse without TRN
Hazards in Landing���Ellipse with TRN
TRN
![Page 11: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/11.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Map-Relative Localizationfor Mars Precision Landing
LVS IMU
LVS Compute Element
Processor Navigation Filter
Data Flow
Virtex 5 FPGA Image Processing Sensor Interfaces
Memory for map
Spacecraft Flight Computer
LVS Camera
Image 1 Image 2
Image 3
IMU
IMU IMU
Image 4 Image 5
IMU
Coarse Landmark Matching Remove Position Error (3km)
Fine Landmark Matching Improve Accuracy (40m)
State Estimation Fuse inertial measurements with landmark
matches and complete in 10 seconds
spacecraft attitude, altitude
map relative ���position
![Page 12: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/12.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyObstacle Detection
• Relevant use cases:– Land vehicles indoors and outdoor
on-road and off-road– Drones: flying and landing– Boats on and under water– Robot manipulators
• Key challenges:– Appearance variability:
• Lighting, weather, season• Surface reflectance, transparency
– Terrain variability– Moving objects– Fail-safe performance
HVU
8 HSMSTs (Teleoperated) 5 USVs
(Autonomous)
• Issues I’ll discuss:– Sensors and phenomenology– 3-D representations– Land, air, and sea examples
![Page 13: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/13.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyStereo Vision in a Marina
![Page 14: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/14.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyStereo + 360-degree Monocular EO/IR
JPL 360(monocular)
JPL Hammerhead (stereo)
Stereo
360
IR Deck
EO Deck Color
![Page 15: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/15.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyUnmanned Surface Vehicles
15
![Page 16: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/16.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyProcessing Blocks
![Page 17: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/17.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
![Page 18: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/18.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Almost crossed à “crossing” rule
not applied USV must give way
1 knot
10 knots
30 knots
10 knots
COLREGS Illustration
Crossing from left Crossing from right Overtaking Head-on
Your boat
Traffic boat
Need more than the geometry to
determine COLREGS situaHons
![Page 19: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/19.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
![Page 20: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/20.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologySeeing through Atmospheric Obscurants
Fog
Smoke from a
controlled burn
Photon 640LWIR camera
Color cameraVisible vs SWIR
in haze
![Page 21: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/21.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Uncooled LWIR Stereo:More Challenging
LBM SAD5 SGBM
![Page 22: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/22.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyNight Operation with Thermal Stereo Cameras
![Page 23: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/23.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyWhy are Negative Obstacles so Hard to Detect?
R
w
H θn
h
α
R
H θp h
α
€
θ p ≈hR
€
θn ≈Hw
R(R + w )
![Page 24: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/24.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Heat Transfer Characteristics:Negative Obstacle Detection
Color crosswise view
Color lengthwise view
MWIR image 1 hr after sundown
Weatherproof sensor enclosure
![Page 25: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/25.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Heat Transfer Characteristics:After Sundown, Holes Cool More Slowly than Surface
• Radiation
• Evapotranspiration (ignored here)
A €
qnegobs1
2negobsq
terrainq
1sideT 2sideT
terrainT
skyT
airTterrainT
terrainq
negobsq
negobsT
C5020 °−=diurnalTterrainT
terrainq
negobsq
negobsT
• Convection
• Conduction
![Page 26: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/26.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Heat Transfer Characteristics:Negative Obstacle Detection
3.8 m
3.7 m
0.53 m
0.53 m
North
9 pm 7 am
9 am 9 pm
5 pm 5 pm
10 pm 7 am
![Page 27: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/27.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyDetection Results using Thermal Signature
Rectified thermal infrared intensity image.
After intensity
difference thresholding.
Closed contours
overlaid on intensity image.
After geometry
based filtering.
Trench 3 pixels wide at first detection. Trench first detected at 18.2m range.
![Page 28: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/28.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyWater Detection: Why is it Useful and is it Hard?
![Page 29: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/29.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Water Body Detection with Reflections in Stereo:Works with Visible and Thermal Images
15:00
100m map
Stereo range imageLeft rectified image
![Page 30: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/30.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologySome Water Detection and Mapping Results
![Page 31: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/31.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyAbsorption Coefficient of Light in Pure Water
![Page 32: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/32.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Foliage Classification andObstacle Detection in Foliage
lhm - 32
![Page 33: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/33.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyPolar Grid Maps
![Page 34: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/34.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
C-space-like obstacle expansion of disparity map
Architecture for obstacle avoidance
Image-based Obstacle Representation
![Page 35: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/35.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
MAV Obstacle Avoidance:Test Site Near JPL
![Page 36: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/36.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
MAV Obstacle Avoidance:Test Results
C-space-like obstacle expansion
of disparity map
Obstacle points and path projected on ground plane
Upright view
![Page 37: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/37.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Research Approach:Challenges and InnovationGetting Very Wide Field of Regard
![Page 38: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/38.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Stereo-OF fusion:
“egocylinder”
• Range from stereo and OF
• Scale propagation from stereo in overlap region
• Projection into common cylinder representation
• C-space expansion in image space
Egocylinder Representation
![Page 39: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/39.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyEgocylinder with Real Data
![Page 40: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/40.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyOnboard Implementation
Update Rates
Stereo 384x240 @ 5 Hz
SFM (LSD-SLAM)
384x240 @ 10 Hzboth cameras
Egocylinder 5 Hz
C-space 5 Hz
Planning5 Hz updates (1 ms
verification, several ms searching)
![Page 41: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/41.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyOnboard Implementation
![Page 42: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/42.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Autonomous Landing:Problems and Solution Characteristics
Gale Crater, Mars
• Problem characteristics • Variable potentially complex 3-D structure • Variable appearance • Variable altitude for approach • Need for very lightweight hardware
• Solution characteristics • Desire dense 3-D perception • Must work from variable altitude • Must work in direct sunlight • Hardware as light as possible – just a camera?
![Page 43: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/43.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Autonomous Safe Landing:Variable Baseline Dense Motion Stereo
![Page 44: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/44.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Onboard Implementationwith Smartphone Processor
![Page 45: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/45.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Autonomous Rooftop Landing of Micro Air Vehicles for Recon Applications
![Page 46: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/46.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Real-time Onboard Mappingfor Safe Landing in Unknown Terrain
Raw image from camera at ~15m height.
Computed elevation map
Landing confidence map (dark blue: highest confidence)
river bed
edge
![Page 47: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/47.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Notional Future Directions for Space Exploration
Pre-Decisional Information – For Planning and Discussion Purposes Only
![Page 48: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/48.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Notional Future Directions for Space Exploration
• Mars sample return• Accessing recurring slope lineae,
caves, and vertical/microgravity
Pre-Decisional Information – For Planning and Discussion Purposes Only
![Page 49: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/49.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Notional Future Directions for Space Exploration
• Mars sample return• Accessing recurring slope lineae• Comet sample return, Ocean Words, Titan
Pre-Decisional Information – For Planning and Discussion Purposes Only
![Page 50: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/50.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Perception and Planning for Robots in Human Environments, Interacting with People
Base reachability Arm
reachability
Desired end-effector goal
Perception and planning for mobile manipulation
Deep learning-based object class labeling/pose estimation; human articulate body pose estimation
Power grasp opportunities
Pinch grasp opportunities
Scene
![Page 51: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/51.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof Technology
Some Thoughts About Back on Earth:Better Perception for Human-Robot Interaction
Integrate facial expressions, head pose, and body pose
into robot perception of people for more intelligent
human robot interaction
S_1 S_2 S_3 S_4 S_5 S_6 S_7 S_8 S_9
S_13S_12 S_15S_14 S_17S_16S_11S_10
Electromyography sleeve with forearm IMU and magnetometer for recognizing arm and
hand gestures
![Page 52: "Using Vision to Enable Autonomous Land, Sea and Air Vehicles," a Keynote Presentation from NASA JPL](https://reader033.fdocuments.us/reader033/viewer/2022051318/587853c21a28ab68198b6cdd/html5/thumbnails/52.jpg)
Jet Propulsion LaboratoryCalifornia Instituteof TechnologyA Few Recent Highlights of Non-NASA Work
Sunset at Gusev Crater, Mars, from Spirit Rover
Questions?