6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007,...
Transcript of 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007,...
![Page 1: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/1.jpg)
6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using
a Monocular Camera
Authors:
Carlos Jaramillo[a]
Ivan Dryanovski[a]
Roberto Valenti[b]
Jizhong Xiao[b]
Presenter: Dr. Jizhong Xiao
City University of New York
The Graduate Center[a] and
The City College of New York[b]
![Page 2: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/2.jpg)
Presentation Outline
1. Problemdescription
2. Existingapproaches
a) MonocularSLAM
b) RGB-DSLAM
3. Proposedmethod
a) Initialposeestimation
b) System’spipeline
4. Results
a) Experiments
b) Performance
5. Futurework
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 2
![Page 3: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/3.jpg)
1. Problem Description
GOAL:6-degree-of-freedom(6-DoF)poselocalization
bysimplyusingamonocularcamera
insidea3Dpoint-clouddensemap
“prebuilt”withdepthsensors
(e.g.,RGB-Dsensor,laserscanner,etc.)
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 3
+
+
![Page 4: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/4.jpg)
1. Problem Description
APPLICATIONEXAMPLES:unconstrainedmotionof
monocularcamerassuchasinsmartphonesormountedin
smallrobots
• Augmentedreality
– Showcases
– Games
–Museumtours
• Mobilerobotnavigation
– Swarmnavigation(SearchandRescue)
1. Aleaderequippedwithpowerfulsensor(s)createsamap
2. Followers(withsimplecameras)localizethemselvesinmap
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 4
http://augmentedpixels.com
Jaramillo’sDREU2009
![Page 5: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/5.jpg)
2. Existing approaches
VisualSLAM:VisualSimultaneous Localization andMapping
a) MonocularSLAM
– MonoSLAM
» [2007,Davisonet.al.]
– PTAM(ParallelTrackingandMapping)
» [2007,Williamset.al.]
– Structurefrommotion(Sfm)
» [1981,Longuet-Higgins]
b) RGB-DSLAM
- Visual3DSLAM
- [2011,Engelhardet.al.]
- Fast3DMapping+VisualOdometry
- [2013,Dryanovski et.al.]
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 5
Resourceintensive:
Needtokeepa
historyoffeatures
inthemap
![Page 6: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/6.jpg)
3. Proposed method
1. Userinitiallymapsoutthescene
(3Ddensepointcloud)
– Avoidsresource-intensive
VisualSLAMtechniques
2. Ourlocalizationmethod:
– Usesdense point-cloud(map)
– Usessingleimagesfroma
monocularcamera
– Wedon’ttrackpoints
– Wegeneratevirtualimages
(usingpreviouspose)ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 6
MONOCULARLOCALIZATIONWITHINA3DMAP
![Page 7: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/7.jpg)
3. Proposed method
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 7
MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)
Initialposeestimation(firsttimeonly!):
1. Inthefirstinputimage,I1,wedetectSURF.Also,extractSURFfromallthemap’s frameimages.
2. WetrainadescriptormatcherfromalltheSURFfeatures.
3. Foreachfeatureintherealimage,wefindn nearestfeatureneighborsusingthematcher.
4. EachfeatureinI1 maypointtoavectorofdescriptormatches.Wetakethetopn candidates
5. TheinitialposeisfoundfromarobustPnPmatchingbetweenthen pointsfromtherealimage
andtheircorresponding3Dpointsinthemapobtainedfromthetopnmatches.
![Page 8: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/8.jpg)
3. Proposed method
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 8
MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)
![Page 9: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/9.jpg)
3. Proposed method
1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 9
MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)
![Page 10: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/10.jpg)
3. Proposed method
1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.
2) 2Dfeaturesarematchedbetweentherealandvirtualimages.
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 10
MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)
![Page 11: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/11.jpg)
3. Proposed method
1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.
2) 2Dfeaturesarematchedbetweentherealandvirtualimages.
3) 2D-to-3Dpointcorrespondencesareobtainedbetweentherealcamera’s2Dfeaturesand
associated3Dpointsinthemap.
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 11
MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)
![Page 12: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/12.jpg)
3. Proposed method
1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.
2) 2Dfeaturesarematchedbetweentherealandvirtualimages.
3) 2D-to-3Dpointcorrespondencesareobtainedbetweentherealcamera’s2Dfeaturesand
associated3Dpointsinthemap.
4) AfterPerspective-n-Point(PnP)+RANSAC,therelative6-DoFtransformationbetweenthereal
andvirtualcamerasisfound.
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 12
MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)
![Page 13: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/13.jpg)
3. Proposed method
1) Thevirtualviewisconstructedbyprojectingthemap’s3Dpointstoaplaneusingthet-1pose.
2) 2Dfeaturesarematchedbetweentherealandvirtualimages.
3) 2D-to-3Dpointcorrespondencesareobtainedbetweentherealcamera’s2Dfeaturesand
associated3Dpointsinthemap.
4) AfterPerspective-n-Point(PnP)+RANSAC,therelative6-DoFtransformationbetweenthereal
andvirtualcamerasisfound.
5) Afinalframetransformationlocalizesthe6-DoFposeofthecamerawithrespecttothemap.
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 13
[World]
CamaxisC
MONOCULARLOCALIZATIONWITHINA3DMAP(Pipeline)
![Page 14: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/14.jpg)
4. Results
Baby’sroomexample(1)
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 14
![Page 15: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/15.jpg)
4. Results
Baby’sroomexample(2)
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 15
![Page 16: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/16.jpg)
4. Results
Baby’sroomexample(3)
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 16
![Page 17: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/17.jpg)
4. Results
Baby’sroomexample(4)
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 17
![Page 18: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/18.jpg)
4. Results
Baby’sroomexample(5)
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 18
![Page 19: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/19.jpg)
4. Results
Officeroomexample(Video)
ROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 19
![Page 20: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/20.jpg)
4. Results
• AtQVGA resolution(320x240pixels),theworst-case
executiontimesrunningona1.7GHzIntelCorei5
processor(insideavirtualmachine)were:
• Bearinmindthatthesetimevaluesincludethe
visualizationoverheadofthe3Dmapandtheimages.
• Intheworstcase,itcanprocess3FPSROBIO2013 6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera 20
Process(Perimageframe) Worst-casetime(ms)
VirtualImageGeneration 70
SURFfeaturedetectionanddescription 100
SURFFeaturematchingwithFLANN 8
PnPwithRANSAC
(1000iters,50inliers,10px reprj.error)
200
Total 378
![Page 21: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/21.jpg)
5. Discussion & Future Work
1. Computingtheinitialposeofthecameraaddsaninitialdelaybefore
theliveimage-feedcanenterthepipeline.
2. Wemustimprovequalityofthevirtualimages
– Affectsthefeaturecorrespondenceprocedure.
3. Improvequalityof3Dmaps
– Virtualimagesdependonmodeldensity(Trymeshedmodels)
4. Wehavetovalidateourmethodbyexperimentingwithbiggermaps
5. Wehavetoperformingerroranalysiswithgroundtruthdatasets.
– Existingdatasetsdon’tproducedensemaps
6. Otherenhancements:
1. AidtherotationestimationwithIMUsensors(phoneshaveit)
2. Usewiderfield-of-viewreal(andvirtual)imagesinorderto
toleratedrasticmotion.
3. Supportdynamicenvironments(onlystaticenvironmentstoday).ROBIO2013 226-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCamera
![Page 22: 6-DoF Pose Localization in 3D Point-Cloud Dense Maps Using a … · 2019-11-26 · » [2007, Williams et. al.] – Structure from motion (Sfm) » [1981, Longuet-Higgins] b) RGB-D](https://reader035.fdocuments.us/reader035/viewer/2022070723/5f01edd97e708231d401bb22/html5/thumbnails/22.jpg)
Thank you!
6-DoFPoseLocalizationin3DPoint-CloudDenseMapsUsingaMonocularCameraROBIO2013 23
Jaramillo,
Carlos
Dryanovski,
Ivan
Valenti,
Roberto
Xiao,
Jizhong