Navion:AFullyIntegratedEnergy-EfficientVisual-Iner:alOdometryAcceleratorforAutonomousNaviga:onofNanoDronesAmrSuleiman,ZhengdongZhang,LucaCarlone,SertacKaraman,andVivienneSze

Demo:TrevorHendersonandDianaWo@

Session/Paper No. : 13.1

Mo:va:on

Virtualandaugmentedrealityonenergy-constraineddevices

SponsorsWanttoknowmore?Visitnavion.mit.edu

NavionResults

KeyChallenges•  Large frame memories for feature

trackingandstereocomputaFon•  Large graph memory with irregular

memoryaccess•  Slowlinearsolver

OurApproach•  Imagecompression

•  4.4xsmallerframememory•  4.9xlesspower

•  Sparsefeaturetracksdatastructure•  5.4xsmallergraphmemory

•  Sparselinearsolver•  5.2xsmallerlinearsolvermemory•  7.2xspeedup

Technology 65nmCMOSChiparea 4.0x5.0mm2

Logicgates 2,043kgatesSRAM 854KBFramesize 480x752Camera 28–171fpsKeyframe 16–90fpsPower 27– 43.2mW

PlaUorm Xeon[2]

ARM[2]

FPGA[2]

ASIC(Thiswork)

Accuracy* 0.20% 0.20% 0.24% 0.28%Cameramaxfps 21 5 20 171Keyframemaxfps 13 3 5 90Power(W) 26.10 2.33 1.46 0.024

NoGPS

Loca:on(Trajectory)

3DMap

UseimagesandinerFaldataforstatees0ma0on

5.0 mm

4.0

mm

Feature Tracking

Feature Detection

Undistort& Rectify

Undistort& Rectify

Sparse StereoLinear Solver

IFE

SharedMemory

Graph

Horizon States

VFE Control

Marginal

References[1]C.Forster,L.Carlone,F.Dellaert,D.Scaramuzza.“On-manifoldpreintegraFontheoryforfastandaccuratevisual-inerFalnavigaFon,”IEEETrans.RoboFcs,2016[2]Z.Zhang,A.Suleiman,L.Carlone,V.Sze,S.Karaman,“Visual-InerFalOdometryonChip:AnAlgorithm-and-HardwareCo-designApproach,”RSS2017.

VisionFrontend(VFE) Backend(BE)FeatureDetec:on(Iden:fyfeatures)•  ExtractHarriscorners(upto200perframe)•  Searchanddistributeacrossthewholeframe•  Computeforeverykeyframe

FeatureTracking(Es:matemovementoffeatures)•  Lucas-KanadepyramidalopFcalflow•  Supportsupto3pyramidlevels•  Computeforallframes(measuremovement

betweenkeyframes)

3DStereo(Es:matedepthoffeatures–a.k.a.landmarks)•  Sparseblockmatching•  Computeforeverykeyframe

FusesvisionandIMUesFmatestorefinethefinalstateesFmates(x’)

Mem

ory

size

3.5 MB 2.4 MB 1.6 MB 854 kB

1.5x 2.6x 4.1x

Image Compression

Two-stage Storage

Sparse LS Memory

Baseline

Frame Graph Linear Solver

Misc.

IMUFrontend(IFE)

AutonomousnavigaFonofnanodrones

x’1

x’2

x’3

Lk

Solvewithstate-of-the-artfactorgraphalgorithm

MinimizeinconsistenciesbetweenmeasurementsacrossFme

Updatestatees0mates(x’)

Over4000+factorsinop0miza0on

IMUFactors VisionFactorst=1

t=2 t=3

*MeasuredonEuRocMAVdataset

VIO:Visual-Iner:alOdometry

Gyroscope: EsFmateRotaFonAccelerometer: EsFmateTranslaFon

Pre-integra:on: IntegrateIMUmeasurementsintooneperkeyframe[1]

NavionChipArchitectureBackend Control

Data & Control BusBuild Graph

Linear Solver

Linearize

Marginal

Retract

GraphLinear Solver

Horizon StatesShared

Memory

Floating Point

Arithmetic

Matrix Operations

Cholesky

Back Substitute

Rodrigues Operations

Feature Tracking

(FT)

Previous FrameLine Buffers

Feature Detection

(FD)

Undistort & Rectify

(UR)

Undistort & Rectify

(UR)

Data & Control Bus

Sparse Stereo (SS)

Vision Frontend Control

RANSAC Fixed Point Arithmetic Point Cloud Pre-IntegrationFloating Point

ArithmeticIMU

memory

Current Frame

Left Frame

Right Frame

Vision Frontend (VFE)

IMU Frontend (IFE)

Backend (BE)

Register File

FullyIntegratedSystem

(nooff-chipprocessingorstorage)

VIOisaspecialinstanceofSimultaneousLocaliza6onandMapping(SLAM)