Artificial Vision-Based Tele-Operation for Lunar Exploration

Students Aaron Roney, Albert Soto, Brian Kuehner, David Taylor, Bonnie Stern,

Nicholas Logan, Stephanie Herd

NASA JSC Mentors Dr. Bob SavelyDr. Mike Goza

Project Mentor Dr. Giovanni Giardini

Project AdvisorProf. Tamás Kalmár-Nagy

Project Members

Electrical Engineering

Computer Engineering

Nuclear Engineering

Mechanical Engineering

Mechanical Engineering

Aerospace Engineering

Aerospace Engineering

Nicholas Logan

Stephanie Herd

Aaron Roney

Albert Soto

Bonnie Stern

Brian Kuehner

David Taylor

Freshman

Freshman

Sophomore

Sophomore

Junior

Senior

Senior

OutlineMotivation and ObjectivesEgo-Motion TheoryCode FlowCalibration and RectificationHardwareTesting ResultsFuture Work

Motivation

Lunar surface exploration

Human perspectiveIn safetyWith low risk

3D environment reconstruction

Self location with artificial vision system

Objectives

Vision SystemEgo-Motion estimationEnvironment reconstruction

Visual Feedback System for Tele-Operations

Tele-Operation SystemRemote control mobile unit

Hardware and Mechanical Implementation

Visual System (onboard the Vehicle)

Ground Station

Vehicle Hardware

Wireless 802.11 Network

Wireless 802.11 Network

Visual System (onboard the Vehicle)

Ground Station

Vehicle Hardware

Wireless 802.11 Network

Wireless 802.11 Network

Ego-Motion Theory

3D Reconstruction Theory

Left image Right image

uleftp uright

p

vleftp vright

p

uleftp

It is impossible to compute the 3D coordinates of an object with a single image

Solution: Stereo Cameras

Disparity computation

3D reconstruction

Image

Disparity map computation: Given 2 images, it is a collection of pixel disparities Point distances can be calculated from disparities

Environment can be reconstructed from disparity map

Left Image Right Image Disparity Map

Environment Reconstruction

Perspective Projection Equation

Main goal: Evaluate the motion (translation and rotation) of the vehicle from sequences of images

Ego-Motion Estimation

Solving will give change in position of the vehicle

Optical Flow Example Optical Flow is related to vehicle movement through the Perspective Projection Equation

Least Square solution

Visual System (onboard the Vehicle)

Ground Station

Vehicle Hardware

Wireless 802.11 Network

Wireless 802.11 Network

Code Flow

Sony VAIO - Pentium 4

Logitech QuickCam Deluxe

Image Processing CodeCalibration ParametersCalibration Parameters

Acquire ImagesAcquire Images

Rectify ImagesRectify Images

Ego-Motion Estimation

Ego-Motion Estimation

Wireless 802.11 Network

Wireless 802.11 Network

Ground StationGround Station

Mobile Unit Detailed Code

Calibration ParametersCalibration Parameters

Snapshot Image Matrix

Snapshot Image Matrix

Image Parameters: Gray Scale (640x480)

…

Image Parameters: Gray Scale (640x480)

…

Acquire Image Rectify Images

Rectified Image Matrix

Rectified Image Matrix

Save ImageSave

Image

T = 0.15 sec T = 0.5 sec

Ego-Motion EstimationEgo-Motion Estimation

Apply Distortion Coefficient to Image Matrix

Apply Distortion Coefficient to Image Matrix

Wireless 802.11 Network

Wireless 802.11 Network

Ground StationGround Station

Ego-Motion Estimation OverviewFind Features

in Right ImageFind Features

in Right Image

Calibration ParametersCalibration Parameters

Right ImageRight Image

Left ImageLeft Image

Track Right Image Features in Left Image

Track Right Image Features in Left Image

New Right Image

New Right Image

New Left Image

New Left Image

Find Features in New Right

Image

Find Features in New Right

Image

Find Features in Left ImageFind Features in Left Image

Find Features in New Left

Image

Find Features in New Left

Image

Track Right Image Features in New

Right Image

Track Right Image Features in New

Right Image

Track Right Image Features in New

Left Image

Track Right Image Features in New

Left Image

Discard All non-Identical Points in

All images

Discard All non-Identical Points in

All images

Displacement Vector

(X, Y, Z, X-Rot, Y-Rot, Z-

Rot)

Displacement Vector

(X, Y, Z, X-Rot, Y-Rot, Z-

Rot)

T = 3 sec

Image Feature Matrix

Image Feature Matrix

Wireless 802.11 NetworkWireless 802.11 Network

Visual System (onboard the Vehicle)

Ground Station

Vehicle Hardware

Wireless 802.11 Network

Wireless 802.11 Network

Calibration and Rectification

Calibration and Rectification

Calibration: Utilizes Matlab tools to determine image distortion associated with the camera

Rectification: Removes the distortion in the images

Visual System (onboard the Vehicle)

Ground Station

Vehicle Hardware

Wireless 802.11 Network

Wireless 802.11 Network

Hardware

Hardware

Mobile Unit

TROPOS Router

Laptop

Web CamerasMobile Unit Base Station

Linksys Router

Operator Computer

Command Computer

Wireless802.11

Wireless802.11

Improvements Implemented in the System

Improved robustness of the softwareImplemented a menu driven system for the operator

using Matlab’s network handling protocolAllowed pictures to be takenRun Ego-motionSending all the results to the operatorGraphic displaying of optical flow

Reduced crashing

Achieved greater mobile unit control

Mobile UnitVehicle Courtesy of Prof. Dezhen Song

Baseline

D

L

FOV1 FOV2

α

Horizontal View

Camera support system3-DOF mechanical neck:

Panoramic rotationTilt rotationTelescopic capability

Controlled height and baseline length

Visual System (onboard the Vehicle)

Ground Station

Vehicle Hardware

Wireless 802.11 Network

Wireless 802.11 Network

Testing Result

Test EnvironmentLight to simulate solar exposure

Black background to eliminate background features

Lunar Environment

Walls to eliminate stray light and side shadows

Measured displacements

Test Setup25 pictures taken from each location (0, 5, 10 and 15

cm) in the Z direction (perpendicular to camera focal plane), unidirectional movementSet 1 25 images located at Z=0Set 2 25 images located at Z=5Set 3 25 images located at Z=10Set 4 25 images located at Z=15

The distances are measured using a tape measureThe cameras are mounted using a semi ridged fixture

Determining the Number of Features

The standard deviation decreases with the more features

But the accuracy of the results decrease

100 Features were selected

Results for 5 cm displacement

Used all 100 images

Compared each set to the previous

Ego-Motion: ExampleOptical Flow Left Image Optical Flow Right Image

RANSACdegree 5 cm Std Dev 10 cm Std Dev 15 cm Std Dev

5 5.31 3.65 9.43 4.39 13.00 3.55

15 4.86 2.09 8.30 3.40 13.65 6.39

30 4.35 1.66 8.21 4.03 15.27 6.01

Problems

Images were not rectifiedPossible motion of cameras between imagesNo image filteringCamera mounting is misalignedImages acquired from the right camera appear

blurry

Conclusions andFuture Work

Demonstrated:Ego-motion estimationEnvironment ReconstructionVehicle control and movementSystem integration

Future Developments:Filtering and improving resultsIncrease the robustness of the vision systemCreate a visual 3D environment map

Thanks to:– Prof. Tamás Kalmár-Nagy– Dr. Giovanni Giardini– Prof. Dezhen Song– Change Young Kim– Magda Lagoudas– Tarek Elgohary– Pedro Davalos

Acknowledgements