PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

Test Plan Review MSL Focused TechnologyInstrument Placement Validation

Test Plan for2D/3D Visual Target Tracking Validation

Won S. KimRobert D. SteeleAdnan I. Ansar

Siqi Chen

August 2, 2004

(818)354-5047Won.S.Kim@jpl.nasa.gov

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

2

Mars Science Laboratory ProjectJet Propulsion Laboratory

Test Plan Objective and Scope

• Test Plan Objective– To test and validate 2D/3D visual target tracking technology– Generate the tracking reliability and error budget model furnished with

experimentally validated numbers

• Scope– Metrology and calibration

• Total station metrology• Mast pan/tilt positioning• Mast and body camera calibration• Mast calibration

– Purely geometric camera handoff with 2D Refinement– Target tracking

• straight path on flat surface• straight path on surface with small rocks• winding path on surface with large rocks• straight-to-target path• hazard avoidance navigation

– Target Tracking using MER images

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

3

Mars Science Laboratory ProjectJet Propulsion Laboratory

System Description

2D/3D Tracking

RoverLocomotorNavigator

Rover PoseEstimator(VisualOdometer)

OptionalCameraHandoff

ActiveCameraControl

NormalizedCrossCorrelation

2D Tracking

Arm’sreach

2D/3D Visual Tracking System

TargetPosition(StereoVision)

ScaleAffineMatching

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

4

Mars Science Laboratory ProjectJet Propulsion Laboratory

• Without visual target tracking

– 3 approach error = 22.2 cm using Pancam and visual odometry (2% error)

• With visual target tracking

– 3 approach error = 1.5 cm at R= 1 m distance using Pancam initially with subsequent camera handoffs to Navcam and Hazcam

Focal length

(1/3” CCDcamera)

Field of view

angles

Stereobaseline

Stereo rangeerror (3)

at 10 mdistance

Target approach error (3) with

2% navigation error

Target approach error

(3) with ideal visual tracking and camera

handoff

16 mm 17° × 13° 30 cm 9.7 cm 22.2 cm 1.5 cm

6 mm 49° × 37° 20 cm 38.8 cm 43.7 cm 3.9 cm

2.3 mm 113° × 86° 10 cm 202.2 cm 203.2 cm 10.1 cm

2 22

2 2 1 1 2

( )s

s s s

fR RR d d

f B f f B

210,

210,10,_ mnavmstereomtrackingno RRR

Computing Target Approach Accuracy

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

5

Mars Science Laboratory ProjectJet Propulsion Laboratory

Baseline Operational Scenario

1. Pancam for 4 m (from 10 m to 6 m)• Minimum stereo range for Pancam is about 5 m

2. Handoff from Pancam to Navcam

3. Navcam for 4m (from 6 m to 2 m)• Viewing angle for Navcam to target gets steep at about 2m

4. Handoff from Navcam to Hazcam

5. Hazcam for 1m (from 2 m to 1 m)• Within arm’s reach

6. Anchor rover and place instrument

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

6

Mars Science Laboratory ProjectJet Propulsion Laboratory

Test Variables and Performance Metric

• Rover motion step size • Straight flat, rocky, or winding path• High-texture or low-texture targets• Lighting conditions• Software algorithms and configuration• Software parameter settings

Tracking performance metrics• Tracking percentage (success rate)• Tracking error

Experimental test variables

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

7

Mars Science Laboratory ProjectJet Propulsion Laboratory

Tracking Reliability & Error Budget Model

Rover locomotion/navigator Rover motion changes the target image, affecting the matching performance: Target image size change Target image roll, pitch, yaw changes

Rover pose estimator using visual odometer (VO)

VO estimation error affects active camera control: Rover pose distance error Rover pose orientation error

Target position estimation using stereo vision

Stereo vision triangulation error affects active camera control: Target position error on image plane

Active camera control to point the fixed-mast to the target (for Pancam and Navcam only)

Fixed-mast pointing errors: pan/tilt encoder resolution pan/tilt backlash mast calibration accuracy

2-D target tracking using normalized cross-correlation, scale, and affine matching

The above active camera control with VO and stereo vision determines the target image displacement, which affects the tracking performance: Tracking success percentage Tracking error

Camera handoff Handoff success percentage Handoff error

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

8

Mars Science Laboratory ProjectJet Propulsion Laboratory

Hypothetical Calculations of Error Budget Model

Terrain Flat Small rocks Large rocks

Approach path straight straight winding

Rover motion step size 20 cm 20 cm 20 cm or 10º

Rover locomotion/navigator Size change per frame Pitch/yaw changes

2% at 10m10% at 2 m

/

2% at 10 m10% at 2 m

10º/

2% at 10m10% at 2m

10º/ 10º

VO rover pose Distance and orientation errors (2%) 0.4 cm / 0.1º 0.4 cm / 0.2º 0.4 cm / 0.3º

Target position error on image plane (stereo triangulation) 1 pixel 1 pixel 1 pixel

Pan/tilt (540:1, 16 CPR) encoder resolution and backlash mast calibration accuracy

0.04º 0.04º 0.04º

Overall orientation error for active camera control 0.1º 0.2º 0.3º

Target image displacement between frames Pancam (17º FOV) Navcam (45º FOV) Hazcam (100º FOV) with active gaze

6 pixels2.3 pixels

1 pixel

12 pixels4.6 pixels2 pixels

18 pixels9.2 pixels3 pixels

2-D target tracking and camera handoff(tracking percentage and error each step)1. Pancam for 4 m (from 10 m to 6 m)2. Handoff from Pancam to Navcam3. Navcam for 4m (from 6 m to 2 m)4. Handoff from Navcam to Hazcam5. Hazcam for 1m (from 2 m to 1 m)

95%; 2 pixels

1 pixel 95%; 3 pixels

1.5 pixels 90%; 2 pixels1 pixel

90%; 3 pixels

1 pixel 90%; 4 pixels

1.5 pixels90%; 2.5 pixels

1 pixel

85%; 4 pixels

1 pixel 85%; 5 pixels

1.5 pixels 85%; 3 pixels1 pixel

Overall single-sol target approach and instrument placement(tracking percentage, pixel error, and placement error)

81%; 3.0 pixels1σ = 2.0 cm3σ = 6.1 cm

73%; 3.5 pixels1σ = 2.4 cm3σ =7.1 cm

61%; 4.0 pixels1σ = 2.7 cm3σ = 8.1 cm

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

9

Mars Science Laboratory ProjectJet Propulsion Laboratory

Test Environment

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

10

Mars Science Laboratory ProjectJet Propulsion Laboratory

Test Environment

• Rocky8 rover in the Mars Yard• Leica TCRA 1103

– 2 mm accuracy for manual tracking; 3 mm for auto tracking

• 10x10 dots calibration target board and target stand• Bricks with reflective tape targets• Camera specifications

Lens manufacturer &focal length

Hor. FOV × Vert. FOV

CCD image resolution

Stereo baseline

Pancam Fujinon; 16 mm 17° × 13° 1024×768 pixels 30 cm

Navcam Raymax; 6 mm 43.5° × 33° 1024×768 pixels 20 cm

Hazcam Computar; 2.3 mm 113° × 86° 640×480 pixels 8.3 cm

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

11

Mars Science Laboratory ProjectJet Propulsion Laboratory

Total Station Metrology

• Prism position data refinement– 1 cm error over 60 cm span for initial rover pose– 1º error in initial heading– 17 cm error over 10 m

• Prism position repeatability– Measure every ¼ turn of prism stick

• Rover pose reference frame precision

TS

TS

TS

TS

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

12

Mars Science Laboratory ProjectJet Propulsion Laboratory

Mast Pan/Tilt Positioning

• Mast Pan/Tilt Zero-Positioning Accuracy– Visual alignment markings

• Mast Pan/Tilt Positioning Repeatability– Move back to zero position from several different initial positions

• Mast Pan/Tilt Positioning Control Resolution– Tiny increment of 0.0005 radians (0.03º)

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

13

Mars Science Laboratory ProjectJet Propulsion Laboratory

Mast and Body Camera Calibration

• Pancam/Navcam/Hazcam– 8 calibration target positions each

– Use acaldots (automatic version of ccaldots) and ccaladj

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

14

Mars Science Laboratory ProjectJet Propulsion Laboratory

Mast and Body Camera Calibration

• Stereo camera error ellipsoids– Bricks with reflective tapes

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

15

Mars Science Laboratory ProjectJet Propulsion Laboratory

Mast Calibration

Move calibration target at different locations with different pan/tilt angles• determine 7 parameters (pan_offset, tilt_offset, x, y, z, thx, thy)• generate camera models relative to masthead• determine mast calibration error• determine reasonably sufficient number of target positions

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

16

Mars Science Laboratory ProjectJet Propulsion Laboratory

Purely Geometric Camera Handoff

• Pancam to Navcam (at about 6 m)• Navcam to Hazcam (at about 2 m)

– measure handoff pixel errors

– analysis based on stereo error ellipsoids

– sensitivity to camera/mast calibration error

From Navcam To Hazcam

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

17

Mars Science Laboratory ProjectJet Propulsion Laboratory

2D Refinement

• Corrects purely geometric camera handoff error by 2D image matching• Template image for the 2nd camera must be generated from the 1st camera• Scaled Template Image

– Uses Field of view angle ratio of Hazcam to Navcam

• Warped Template Image– Uses camera Models and stereo range maps of Navcam and Hazcam

Template image reconstructed from Navcam Actual Hazcam image

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

18

Mars Science Laboratory ProjectJet Propulsion Laboratory

2-D Refinement

Overlay of Navcam image on Hazcam image

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

19

Mars Science Laboratory ProjectJet Propulsion Laboratory

Target Tracking along Straight Approach Paths on Flat Surface

Target Tracking Validation ExperimentsTarget Tracking Validation Experiments– Furnish the tracking reliability and error budget model with

experimentally validated numbers – If some functionalities are not available, conduct portions of the

baseline operations

Rover Locomotion/Navigator– Linear steps from 10 m to 1 m

• 0.25 m – 36 steps (10, 9.75, 9.5, 9.25, 9, …, 2, 1.75, 1.5, 1.25, 1)

• 0.5 m – 18 steps (10, 9.5, 9, …, 2, 1.5, 1)

• 1 m – 9 steps (10, 9, …, 2, 1)

– Percent-change steps from 10 m to 1m• 10% change – 22 steps (10, 9, 8.1, 7.29, 6.56, …, 1.21, 1.09, 0.99)

• 20% change – 11 steps (10, 8, 6.4, 5.1, 4.1, …, 1.7, 1.3, 1.1, 0.9)

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

20

Mars Science Laboratory ProjectJet Propulsion Laboratory

Pose Estimator, Stereo Localization, andMast Pointing

Pose Estimator– Wheel odometer + IMU + Visual odometer

– Measure pose estimation inaccuracy by comparing with total station metrology

Stereo Localization– Use stereo error ellipsoids data

– Compare with total station metrology

Mast Pan/Tilt Pointing– Select an initial target at different image positions

– Run the tracker one step without rover motion first• zero rover motion excludes pose estimation error

– After the mast completes the new pointing, measure discrepancy between target image position and image center (512, 384)

– Repeat the above with rover motion (to include pose estimation error)

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

21

Mars Science Laboratory ProjectJet Propulsion Laboratory

Normalized Cross-Correlation andScale/Affine Matching

• Refer to 2D Target Tracking Validation Report • Conduct actual tracking runs as well as off-line tracking runs by using

the image sets of actual tracking runs– use different target selections and image skips

– measure the tracking reliability and error

• Linear or percent-change step size? What size?• Target loss: how to prevent it or how to detect it?• Effect of lighting

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

22

Mars Science Laboratory ProjectJet Propulsion Laboratory

Camera Handoff and Hazcam Tracking

Camera handoff– Examine handoff errors for the tracking runs under the baseline

operational scenario (Pancam to Navcam; Navcam to Hazcam)

– Compare with earlier experimental data tested separately

Hazcam Tracking– Without active camera control– Estimate the new target image position based on rover pose

estimator, stereo triangulation, and camera model– Examine the initial error of the estimated target image position,

and tracking success rate and error after 2-D template matching

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

23

Mars Science Laboratory ProjectJet Propulsion Laboratory

Target Tracking Tests with Different Approach Paths

• Target tracking along straight approach paths on surface with small rocks

• Target tracking along winding approach paths on surface with large rocks

• Target tracking with heading-towards-target Path• Target tracking with hazard avoidance navigation• Target tracking using MER images

August 2, 2004 PRE-DECISIONAL DRAFT: For Planning and Discussion Purposes Only

24

Mars Science Laboratory ProjectJet Propulsion Laboratory

Software Modifications Desired for Test Plan

1. Implement exact mast inverse kinematics– camera frames are not in general parallel to the masthead frame– CAHVOR model optical axis does not in general pass through the image center

2. Add parameters for params.txt– Window size for VO– Window size for tracking– Image skip for off-line runs

3. Add two initial search options to the pgm header– Camera pose– Image position

4. Hazcam tracking without active camera control5. Camera handoff

– Pancam to Navcam– Navcam to Hazcam– 2-D Refinement

6. Add parameters for drive.txt– Different step size for VO??

7. Lost target detection8. Stereo triangulation instead of full stereo image processing9. Clean-up of output file format